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• 

The OECD Programme for International Student Assessment (PISA), created in 1997, represents a commitment 
by the governments of OECD member countries to monitor the outcomes of education systems in terms of 
student achievement, within a common internationally agreed framework. PISA is a collaborative effort, bringing 
together scientific expertise from the participating countries and steered jointly by their governments on the basis 
of shared, policy-driven interests. Participating countries take responsibility for the project at the policy level. 
Experts from participating countries also serve on working groups that are charged with linking the PISA policy 
objectives with the best available substantive and technical expertise in the field of internationally comparative 
assessment. Through involvement in these expert groups, countries ensure that the PISA assessment instruments 
are internationally valid and take into account the cultural and curricular context of OECD member countries. 
They also have strong measurement properties, and place an emphasis on authenticity and educational validity. 

PISA 2009 represents a continuation of the data strategy adopted in 1997 by OECD countries. As in 2000, 
reading literacy is the focus of the PISA 2009 survey, but the reading framework has been updated and now 
also includes the assessment of reading of electronic texts. The framework for assessing mathematics was 
fully developed for the PISA 2003 assessment and remained unchanged in 2009. Similarly, the framework for 
assessing science was fully developed for the PISA 2006 assessment and remained unchanged in 2009. 

This publication presents the guiding principles of the PISA 2009 assessment, which are described in terms 
of the skills students need to acquire, the processes that need to be performed and the contexts in which 
knowledge and skills are applied. Eurther, it illustrates the assessment domains with a range of sample tasks. 
These have been developed by expert panels under the direction of Raymond Adams, Juliette Mendelovits, 
Ross Turner and Barry McCrae from the Australian Council for Educational Research (ACER) and 
Henk Moelands (CITO). The reading expert group was chaired by Irwin Kirsch of Educational Testing Service in 
the United States. The mathematics expert group was chaired by Jan de Lange of the University of Utrecht in the 
Netherlands and the science expert group was chaired by Rodger Bybee of the Biological Science Curriculum 
Study in the United States. The questionnaire expert group was chaired by Jaap Scheerens of University of 
Twente in the Netherlands. The members of the expert groups are listed in Annex C of this publication. The 
frameworks have also been reviewed by expert panels in each of the participating countries. The chapters on 
reading, mathematics and science were drafted by the respective expert groups under the direction of their 
chairs, Irwin Kirsch (reading), Jan de Lange (mathematics) and Rodger Bybee (science). The chapter on the 
questionnaire framework was drafted by Henry Levin of Teachers College, Columbia University, New York, and 
is based on a review of central issues, addressed in conceptual papers for the PISA Governing Board, prepared 
by Jaap Scheerens in collaboration with the questionnaire expert group. The publication was prepared by the 
OECD Secretariat, principally by Andreas Schleicher, Karin Zimmer, Juliet Evans and Niccolina Clements. 

The report is published on the responsibility of the Secretary-General of the OECD. 
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Executive Summ^f-y 

• 

Parents, students, teachers, governments and the general public - all stakeholders - need to know how well 
their education systems prepare students for real-life situations. Many countries monitor students' learning to 
evaluate this. Comparative international assessments can extend and enrich the national picture by providing 
a larger context within which to interpret national performance. They can show what is possible in education, 
in terms of the quality of educational outcomes as well as in terms of equity in the distribution of learning 
opportunities. They can support setting policy targets by establishing measurable goals achieved by other 
systems and help to build trajectories for reform. They can also help countries work out their relative strengths 
and weaknesses and monitor progress. 

In response to the need for cross-nationally comparable evidence on student performance, the Organisation for 
Economic Co-operation and Development (OECD) launched the OECD Programme for International Student 
Assessment (PISA) in 1 997. PISA represents a commitment by governments to monitor the outcomes of education 
systems through measuring student achievement on a regular basis and within an internationally agreed common 
framework. It aims to provide a new basis for policy dialogue and for collaboration in defining and implementing 
educational goals, in innovative ways that reflect judgements about the skills that are relevant to adult life. 

PISA is a collaborative effort undertaken by its participants - the OECD member countries as well as over 
30 non-member partner economies - to measure how well students, at age 15, are prepared to meet the 
challenges they may encounter in future life. Age 1 5 is chosen because at this age students are approaching the 
end of compulsory education in most OECD countries. PISA, jointly guided by the participating governments, 
brings together the policy interests of countries with scientific expertise at both national and international 
levels. PISA has been measuring the knowledge, skills and attitudes of 1 5-year-olds over the last ten years and 
is therefore able to give some insight into how countries are faring over time. 

The PISA assessment takes a broad approach to measuring knowledge, skills and attitudes that reflect current 
changes in curricula, moving beyond the school-based approach towards the use of knowledge in everyday 
tasks and challenges. It is based on a dynamic model of lifelong learning in which new knowledge and skills 
necessary for successful adaptation to a changing world are continuously acquired throughout life. PISA focuses 
on things that 1 5-year-old students will need in the future and seeks to assess what they can do with what they 
have learned - reflecting the ability of students to continue learning throughout their lives by applying what they 
learn in school to non-school environments, evaluating their choices and making decisions. The assessment is 
informed, but not constrained, by the common denominator of national curricula. Thus, while it does assess 
students' knowledge, PISA also examines their ability to reflect, and to apply their knowledge and experience 
to real-life issues. For example, in order to understand and evaluate scientific advice on food safety an adult 
would need not only to know some basic facts about the composition of nutrients, but also to be able to apply 
that information. The term literacy is used to encapsulate this broader concept of knowledge and skills, and 
the PISA assessment aims to determine the extent to which 1 5-year-old students can activate various cognitive 
processes that would enable them to make effective use of the reading, mathematical and scientific knowledge 
and skills they have acquired throughout their schooling and related learning experiences up to that point. 

PISA is designed to collect information through three-yearly assessments and presents data on domain-specific 
knowledge and skills in reading, mathematics and science of students, schools and countries. It combines 
the assessment of science, mathematics and reading with information on students' home background, their 
approaches to learning, their learning environments and their familiarity with computers. Student outcomes are 
then associated with these background factors. Thereby, PISA provides insights into the factors that influence the 
development of skills and attitudes at home and at school, and examines how these factors interact and what 
the implications are for policy development. 
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PISA uses: 1) strong quality assurance mechanisms for translation, sampling and test administration; 2) measures 
to achieve cultural and linguistic breadth in the assessment materials, particularly through countries' participation 
in the development and revision processes for the production of the items; and 3) state of the art technology and 
methodology for data handling. The combination of these measures produces high quality instruments and outcomes 
with superior levels of validity and reliability to improve the understanding of education systems as well as students' 
knowledge, skills and attitudes. 

This publication presents the theory underlying the PISA 2009 assessment, including a re-developed and 
expanded framework for reading literacy, which incorporates an innovative component on the capacity to 
read and understand electronic texts, thus reflecting the importance of information and computer technologies 
in modern societies. It also provides the basis for the assessment of mathematics and science. Within each 
domain, the knowledge content that students need to acquire is defined, as well as the processes that need to 
be performed and the contexts in which knowledge and skills are applied. It also illustrates the domains and 
their aspects with sample tasks. Finally, the theory underlying the context questionnaires is presented. These are 
used to gather information from students, schools and parents on the students' home background and attitudes, 
their learning histories and their learning environments at school. 



Box A ■ What is PISA? 



Basics 

■ An internationally standardised assessment that was jointly developed by participating countries 
and administered to 15-year-olds in educational programmes. 

■ A survey implemented in 43 countries and economies in the first cycle (32 in 2000 and 1 1 in 2002), 
41 in the second cycle (2003), 57 in the third cycle (2006) and 67 in the fourth cycle (2009). 

■ The test is typically administered to between 4 500 and 10 000 students in each country/economy. 

Content 

■ PISA 2009 covers the domains of reading, mathematics and science not merely in terms of 
whether students can reproduce specific subject matter knowledge, but also whether they can 
extrapolate from what they have learned and apply their knowledge in novel situations. 

■ Emphasis is on the mastery of processes, the understanding of concepts and the ability to 
function in various situations within each domain. 

Methods 

■ Paper-and-pencil tests are used, with assessments lasting a total of two hours for each student. 
In a range of countries and economies, an additional 40 minutes are devoted to the assessment 
of reading and understanding electronic texts. 

■ Test items are a mixture of multiple-choice items and questions requiring students to construct 
their own responses. The items are organised in groups based on a passage setting out a real-life 
situation. 

■ A total of about 390 minutes of test items is covered, with different students taking different 
combinations of test items. 



PISA 2009 ASSESSMENT FRAMEWORK - KEY COMPETENCIES IN READING, MATHEMATICS AND SCIENCE O OECD 2009 



INTRODUCTION 



■ Students answer a background questionnaire, which takes 30 minutes to complete, providing 
information about themselves and their homes. School principals are given a 20-minute 
questionnaire about their schools. In some countries and economies, optional short questionnaires 
are administered to: 1) parents to provide further information on past and present reading 
engagement at the students' homes; and 2) students to provide information on their access to and 
use of computers as well as their educational history and aspirations. 

Assessment cycle 

■ The assessment takes place every three years with a strategic plan in place extending through 
to 201 5. 

■ Each of these cycles looks in depth at a major domain, to which two-thirds of testing time is 
devoted; the other domains provide a summary profile of skills. Major domains have been 
reading in 2000, mathematics in 2003 and science in 2006. In 2009, the major domain is again 
reading literacy. 

Outcomes 

■ A basic profile of knowledge and skills among 1 5-year-old students. 

■ Contextual indicators relating results to student and school characteristics. Trend indicators 
showing how results change over time. 

■ A valuable knowledge base for policy analysis and research. 



BASIC FEATURES OF PISA 2009 

PISA 2009 is the fourth cycle of a data strategy defined in 1 997 by participating countries. The publications 
Measuring Student Knowledge and Skills A New Framework for Assessment (OECD, 1 999), The PISA 
2003 Assessment Framework Mathematics, Reading, Science and Problem Solving Knowledge and Skills 
(OECD, 2003) and Assessing Scientific, Reading and Mathematical Literacy - A Framework for PISA 2006 
(OECD, 2006) presented the conceptual framework underlying the first three cycles of PISA. The results from 
those cycles were presented in the publications Knowledge and Skills for Life - First Results from PISA 2000 
(OECD, 2001), Learning for Tomorrow's World: First Results from PISA 2003 (OECD, 2004) and PISA 2006: 
Science Competencies for Tomorrow's World (OECD, 2007). All publications are also available on the PISA 
website: www.pisa.oecd.org. The results allow national policy makers to compare the performance of their 
education systems with those of other countries. Similar to the previous assessments, the 2009 assessment 
covers reading, mathematics and science, with the major focus on reading literacy. Students also respond to 
a background questionnaire, and additional supporting information is gathered from the school authorities. 
In 14 countries and economies information is also gathered from the students' parents. Sixty-seven countries 
and economies, including all 30 OECD member countries, are taking part in the PISA 2009 assessment. 
Together, they comprise almost 90% of the world's economy. 

Since the aim of PISA is to assess the cumulative yield of education systems at an age where compulsory 
schooling is still largely universal, testing focuses on 15-year-olds enrolled in both school-based and work- 
based educational programmes. Between 4 500 and 10 000 students from at least 150 schools are typically 
tested in each country, providing a good sampling base from which to break down the results according to a 
range of student characteristics. 
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The primary aim of the PISA assessment is to determine the extent to which young people have acquired 
the wider knowledge and skills in reading, mathematics and science that they will need in adult life. The 
assessment of cross-curricular competencies continues to be an integral part of PISA 2009. The main reasons 
for this broadly oriented approach are: 



■ Although specific knowledge acquisition is important in school learning, the application of that knowledge 
in adult life depends crucially on the acquisition of broader concepts and skills. In reading, the capacity to 
develop interpretations of written material and to reflect on the content and qualities of text are central skills. 
In mathematics, being able to reason quantitatively and to represent relationships or dependencies is more 
relevant than the ability to answer familiar textbook questions when it comes to deploying mathematical 
skills in everyday life. In science, having specific knowledge, such as the names of plants and animals, is of 
less value than understanding broad topics such as energy consumption, biodiversity and human health in 
thinking about the issues under debate in the adult community. 

■ In an international setting, a focus on curriculum content would restrict attention to curriculum elements 
common to all or most countries. This would force many compromises and result in an assessment too 
narrow to be of value for governments wishing to learn about the strengths and innovations in the education 
systems of other countries. 

■ Certain broad, general skills are essential for students to develop. They include communication, adaptability, 
flexibility, problem solving and the use of information technologies. These skills are developed across the 
curriculum and an assessment of them requires a broad cross-curricular focus. 

PISA is not a single cross-national assessment of the reading, mathematics and science skills of 15-year-old 
students. It is an ongoing programme that, over the longer term, will lead to the development of a body of 
information for monitoring trends in the knowledge and skills of students in various countries as well as in 
different demographic subgroups of each country. On each occasion, one domain is tested in detail, taking 
up nearly two-thirds of the total testing time. This data collection strategy provides a thorough analysis of 
achievement in each area every nine years and a trend analysis every three. The major domain was reading 
in 2000, mathematics in 2003 and science in 2006. In 2009, it is reading again, building on a modified 
reading framework which incorporates the reading of electronic texts and elaborates the constructs of reading 
engagement and meta-cognition (see Chapter 1). The mathematics and science frameworks for PISA 2009 are 
the same as for the previous assessment (see Chapters 2 and 3 respectively). 



Similar to previous PISA cycles, the total time spent on the PISA 2009 tests by each student is two hours, 
but information is obtained from about 390 minutes worth of test items. For each country, the total set of 
questions is packaged into 13 linked testing booklets. Each booklet is taken by a sufficient number of students 
for appropriate estimates to be made of the achievement levels on all items by students in each country and in 
relevant sub-groups within a country (such as boys and girls, and students from different social and economic 
contexts). Students also spend 30 minutes answering a background questionnaire. In addition to this core 
assessment, in a range of countries and economies, the assessment includes a computerised test on the reading 
and understanding of electronic texts. 



The PISA assessment provides three main types of outcomes: 

■ Basic indicators that provide a baseline profile of the knowledge and skills of students. 

■ Contextual indicators that show how such skills relate to important demographic, social, economic and 
educational variables. 

■ Indicators on trends that emerge from the on-going nature of the data collection and that show changes in 
outcome levels and distributions, and in relationships between student-level and school-level background 
variables and outcomes. 



Although indicators are an adequate means of drawing attention to important issues, they do not provide 
answers to policy questions. PISA has therefore also developed a policy-oriented analysis plan that goes beyond 
the reporting of indicators. 
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WHAT MAKES PISA UNIQUE 

PISA focuses on young people's ability to use their knowledge and skills to meet real-life challenges. This 
orientation reflects a change in the goals and objectives of curricula themselves, which are increasingly 
concerned with what students can do with what they learn at school and not merely with whether they have 
mastered specific curricular content. 

Key features driving the development of PISA have been its: 

■ Policy orientation, which connects data on student learning outcomes with data on students' characteristics 
and on key factors shaping their learning inside and outside school in order to draw attention to differences 
in performance patterns and to identify the characteristics of schools and education systems that have high 
performance standards. 

■ Innovative literacy concept, which is concerned with the capacity of students to apply knowledge and skills 
in key subject areas and to analyse, reason and communicate effectively as they pose, solve and interpret 
problems in a variety of situations. 

■ Relevance to lifelong learning, which does not limit PISA to assessing students' curricular and cross-curricular 
competencies, but also asks them to report on their own motivation to learn, their beliefs about themselves 
and their learning strategies. 

■ Regularity, which enables countries to monitor their progress in meeting key learning objectives. 

■ Breadth of geographical coverage and collaborative nature, which in PISA 2009 encompasses the 30 OECD 
member countries and over 30 partner countries and economies. 

The relevance of the knowledge and skills measured by PISA is confirmed by recent studies tracking young people 
in the years after they have been assessed by PISA. Studies in Australia, Canada and Denmark display a strong 
relationship between the performance in reading on the PISA 2000 assessment at age 15 and the chance of a student 
completing secondary school and of carrying on with post-secondary studies at age 19. For example, Canadian 
students who had achieved reading proficiency Level 5 at age 1 5 were 1 6 times more likely to be enrolled in post- 
secondary studies when they were 1 9 years old than those who had not reached the reading proficiency Level 1 . 

PISA is the most comprehensive and rigorous international programme to assess student performance and to 
collect data on the student, family and institutional factors that can help to explain differences in performance. 
Decisions about the scope and nature of the assessments and the background information to be collected are 
made by leading experts in participating countries, and are steered jointly by governments on the basis of shared, 
policy-driven interests. Substantial efforts and resources are devoted to achieving cultural and linguistic breadth 
and balance in the assessment materials. Stringent quality assurance mechanisms are applied in translation, 
sampling and data collection. As a consequence, the results of PISA have a high degree of validity and reliability, 
and can significantly improve understanding of the outcomes of education in the world's economically most 
developed countries, as well as in a growing number of countries at earlier stages of economic development. 

Across the world, policy makers are using PISA findings to: gauge the knowledge and ski I Is of students in their own country 
in comparison with those of the other participating countries; establish benchmarks for educational improvement, for 
example, in terms of the mean scores achieved by other countries or their capacity to provide high levels of equity in 
educational outcomes and opportunities; and understand relative strengths and weaknesses of their education systems. 
The interest in PISA is illustrated by the many reports produced in participating countries, the numerous references to 
the results of PISA in public debates and the intense media attention shown to PISA throughout the world. 



AN OVERVIEW OF WHAT IS BEING ASSESSED IN EACH DOMAIN 

Box B presents a definition of the three domains assessed in PISA 2009. The definitions all emphasise functional 
knowledge and skills that allow one to participate actively in society. Such participation requires more than just 
being able to carry out tasks imposed externally by, for example, an employer. It also means being equipped 
to take part in decision-making processes. In the more complex tasks in PISA, students are asked to reflect on 
and evaluate material, not just to answer questions that have single correct answers. The definitions address the 
capacity of students to extrapolate from what they have learned, and to apply their knowledge in novel settings. 
The definitions also focus on the students' capacity to analyse, reason and communicate effectively, as they 
pose, solve and interpret problems in a variety of situations. 
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Box B ■ Definitions of the domains 



Reading literacy: An individual's capacity to: understand, use, reflect on and engage with 
written texts, in order to achieve one's goals, to develop one's knowledge and potential, and to 
participate in society. 

Mathematical literacy: An individual's capacity to identify and understand the role that 
mathematics plays in the world, to make well-founded judgements and to use and engage with 
mathematics in ways that meet the needs of that individual's life as a constructive, concerned 
and reflective citizen. 

Scientific literacy: An individual's scientific knowledge and use of that knowledge to identify 
questions, to acquire new knowledge, to explain scientific phenomena, and to draw evidence- 
based conclusions about science-related issues, understanding of the characteristic features of 
science as a form of human knowledge and enquiry, awareness of how science and technology 
shape our material, intellectual, and cultural environments, and willingness to engage in 
science-related issues, and with the ideas of science, as a reflective citizen. 



Reading literacy (elaborated in Chapter 1 ) is defined in terms of students' ability to understand, use and reflect 
on written text to achieve their purposes. This aspect of literacy has been well established by previous surveys 
such as the International Adult Literacy Survey (lALS), but is taken further in PISA by the introduction of an 
active element - the capacity not just to understand a text but to reflect on it, drawing on one's own thoughts 
and experiences. In PISA, reading literacy is assessed in relation to the: 

■ Text format: Often students' reading assessments have focused on continuous texts or prose organised in 
sentences and paragraphs. From its inception, PISA has used in addition non-continuous texts that present 
information in other ways, such as in lists, forms, graphs, or diagrams. It has also distinguished between 
a range of prose forms, such as narration, exposition and argumentation. In PISA 2009, the framework 
encompasses both print and electronic texts, and the distinctions outlined above are applied to both. These 
distinctions are based on the principle that individuals will encounter a range of written material in their civic 
and work-related adult life (e.g. application, forms, advertisements) and that it is not sufficient to be able to 
read a limited number of types of text typically encountered in school. 

■ Reading processes (aspects): Students are not assessed on the most basic reading skills, as it is assumed that 
most 1 5-year-old students will have acquired these. Rather, they are expected to demonstrate their proficiency 
in accessing and retrieving information, forming a broad general understanding of the text, interpreting it, 
reflecting on its contents and reflecting on its form and features. 

■ Situations: These are defined by the use for which the text was constructed. For example, a novel, personal 
letter or biography is written for people's personal use; official documents or announcements for public use; 
a manual or report for occupational use; and a textbook or worksheet for educational use. Since some groups 
may perform better in one reading situation than in another, it is desirable to include a range of types of 
reading in the assessment items. 

Mathematical literacy (elaborated in Chapter 2) is concerned with the ability of students to analyse, reason, and 
communicate ideas effectively as they pose, formulate, solve, and interpret solutions to mathematical problems 
in a variety of situations. The PISA mathematics assessment has, so far, been designed in relation to the: 

■ Mathematical content: This is defined mainly in terms of four overarching ideas (quantity, space and shape, 
change and relationships, and uncertainty) and only secondarily in relation to curricular strands (such as 
numbers, algebra and geometry). 
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■ Mathematical processes: These are defined by individual mathematical competencies. These include the use 
of mathematical language, modelling and problem-solving skills. Such skills, however, are not separated 
out in different test items, since it is assumed that a range of competencies will be needed to perform any 
given mathematical task. Rather, questions are organised in terms of competency clusters defining the type 
of thinking skill needed. 

■ Situations: These are defined in terms of the ones in which mathematics is used, based on their distance 
from the students. The framework identifies five situations; personal, educational, occupational, public and 
scientific. 

However, a major revision of the PISA mathematics framework is currently underway in preparation for the 

PISA 201 2 assessment. 

Scientific literacy (elaborated in Chapter 3) is defined as the ability to use scientific knowledge and processes not 

only to understand the natural world but to participate in decisions that affect it. The PISA science assessment 

is designed in relation to: 

■ Scientific knowledge or concepts: These constitute the links that aid understanding of related phenomena. In 
PISA, while the concepts are the familiar ones relating to physics, chemistry, biological sciences and earth 
and space sciences, they are applied to the content of the items and not just recalled. 

m Scientific processes: These are centred on the ability to acquire, interpret and act upon evidence. Three 
such processes present in PISA relate to: 1) describing, explaining and predicting scientific phenomena, 
2) understanding scientific investigation, and 3) interpreting scientific evidence and conclusions. 

m Situations or contexts: These concern the application of scientific knowledge and the use of scientific 
processes applied. The framework identifies three main areas: science in life and health, science in Earth and 
environment, and science in technology. 



ASSESSING AND REPORTING PISA 2009 

Similar to the previous assessments in PISA, the assessment in 2009 mainly consists of pencil and paper 
instruments. In addition, a computerised assessment of reading of electronic texts is carried out in a range 
of countries and economies. Both the paper-and-pencil assessment and the computer-based assessment 
include a variety of types of questions. Some require students to select or produce simple responses that can 
be directly compared with a single correct answer, such as multiple-choice or closed-constructed response 
items. These questions have either a correct or incorrect answer and often assess lower-order skills. Others are 
more constructive, requiring students to develop their own responses designed to measure broader constructs 
than those captured by more traditional surveys, allowing for a wider range of acceptable responses and more 
complex marking that can include partially correct responses. 

Not all students answer all questions in the assessment. For the paper-and-pencil assessment of reading, 
mathematics and science, the PISA 2009 test units are arranged in 13 clusters, with each cluster designed to 
occupy 30 minutes of testing time. In each country, there are seven reading clusters, three mathematics clusters 
and three science clusters. The clusters are placed in 13 booklets, according to a rotated test design. Each 
booklet contains four clusters and each student is assigned one of these two-hour booklets. There is at least one 
reading cluster in each booklet. 

For the assessment of reading, two alternative sets of booklets are provided in PISA 2009, from which a country 
will implement one. One set of booklets comprises items distributed across a range of difficulty similar to that of 
previous cycles. The second set also contains items covering the full range of difficulty, but includes more items 
at the easier end of the range, in order to obtain better descriptive information about what students at the lower 
end of the ability spectrum know, understand and can do as readers. All participating countries and economies 
administer 11 common clusters: five clusters of reading items, three clusters of mathematics items and three 
clusters of science items. In addition, countries administer one of two alternative pairs of reading clusters. The 
performance of students in all participating countries and economies will be represented on a common reading 
literacy scale. 
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In a range of countries and economies, the reading and understanding of electronic texts is assessed in a 
^ 40-minute test. The test units are arranged in six clusters of 20 minutes each. Two clusters are placed in a 
^ booklet, according to a rotated design, so the test material consists of six booklets with two clusters each. Every 
student taking part in the computer-based assessment is given one of the six booklets to work on. For the paper- 
and-pencil assessment as well as the computerised assessment, knowledge and skills are assessed through units 
consisting of a stimulus (e.g. text, table, chart, figures, etc.) followed by a number of tasks associated with this 
common stimulus. This is an important feature, allowing questions to go into greater depth than if each question 
were to introduce a wholly new context. It allows time for the student to digest material that can then be used 
to assess multiple aspects of performance. 

Results from PISA have been reported using scales with an average score of 500 and a standard deviation of 
100 for all three domains, which means that two-thirds of students across OECD countries scored between 
400 and 600 points. These scores represent degrees of proficiency in a particular domain. Reading literacy was 
the major domain in 2000, and the reading scales were divided into five levels of knowledge and skills. The 
main advantage of this approach is that it describes what students can do by associating the tasks with levels of 
difficulty. Additionally, results were also presented through three subscales of reading: retrieving information, 
interpreting texts, and reflection and evaluation. A proficiency scale was also available for mathematics and 
science, though without levels therefore recognising the limitation of the data from minor domains. PISA 2003 
built upon this approach by specifying six proficiency levels for the mathematics scale, following a similar 
approach to what was done in reading. There were four subscales in mathematics: space and shape, change 
and relationships, quantity, and uncertainty. In a similar manner, the reporting oi science in PISA 2006 specified 
six proficiency levels for the science scale. The three subscales in science related to identifying scientific 
issues, explaining phenomena scientifically and using scientific evidence. Additionally, country performance 
was compared on the bases of knowledge about science and knowledge of science. The three main areas of 
knowledge of science were physical systems, living systems and earth and space systems. PISA 2009 will be the 
first time that reading literacy will be re-assessed as a major domain, and will provide trend results for all three 
domains of reading, mathematics and science. 



THE CONTEXT QUESTIONNAIRES AND THEIR USE 

To gather contextual information, PISA asks students and the principals of their schools to respond to background 

questionnaires of around 30 minutes in length. These questionnaires are central to the analysis of results in 

terms of a range of student and school characteristics. Chapter 4 presents the questionnaire framework in detail. 

The questionnaires from all assessments (PISA 2000, 2003, 2006 and 2009) are available on the PISA website: 

www.pisa.oecd.org. The questionnaires seek information about: 

■ Students and their family backgrounds, including their economic, social and cultural capital. 

■ Aspects of students' lives, such as their attitudes towards learning, their habits and life inside school, and their 
family environment. 

■ Aspects of schools, such as the quality of the schools' human and material resources, public and private 
control and funding, decision-making processes, staffing practices and the school's curricular emphasis and 
extra-curricular activities offered. 

■ Context of instruction, including institutional structures and types, class size, classroom and school climate 
and reading activities in class. 

■ Aspects of learning and instruction in reading, including students' interest, motivation and engagement. 

Three additional questionnaires are offered as international options: 

■ A computer familiarity questionnaire focusing on the availability and use of information and communications 
technology (ICT), including where ICT is mostly used, as well as on the students' ability to carry out computer 
tasks and their attitudes towards computer use. The OECD published a report resulting from analysis of data 
collected via this questionnaire in 2003: Are Students Ready for a Technology-Rich World? What PISA Studies 
Tell Us (OECD, 2005). As part of its New Millennium Learners project, the OECD's Centre for Educational 
Research and Innovation (CERI) will be publishing a similar report using the PISA 2006 data. 

■ An educational career questionnaire collecting additional information on interruptions of schooling and 
changes of schools, expected educational attainment and lessons or tutoring outside of school. 
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■ A parent questionnaire focusing on a number of topics including the student's past reading engagement, the 
parents' own reading engagement, home reading resources and support, and the parents' perceptions of and 
involvement in their child's school. 

The contextual information collected through the student and school questionnaires, as well as the optional 
computer familiarity, educational career and parent questionnaires, comprises only a part of the total amount 
of information available to PISA. Indicators describing the general structure of the education systems (their 
demographic and economic contexts - for example, costs, enrolments, school and teacher characteristics, and 
some classroom processes) and their effect on labour market outcomes are already routinely developed and 
applied by the OECD (e.g. the yearly OECD publication of Education at a Glance). 



COLLABORATIVE DEVELOPMENT OF PISA AND ITS ASSESSMENT FRAMEWORK 

PISA represents a collaborative effort among the OECD member governments to provide a new kind of 
assessment of student achievement on a recurring basis. The assessments are developed co-operatively, agreed 
by participating countries, and implemented by national organisations. The constructive co-operation of 
students, teachers and principals in participating schools has been crucial to the success of PISA during all 
stages of the development and implementation. 

The PISA Governing Board (PGB), representing all nations at the senior policy levels, determines the policy 
priorities for PISA in the context of OECD objectives and oversees adherence to these priorities during the 
implementation of the programme. This includes setting priorities for the development of indicators, for the 
establishment of the assessment instruments and for the reporting of the results. Experts from participating 
countries also serve on working groups charged with linking the PISA policy objectives with the best 
internationally available technical expertise in the different assessment domains. By participating in these 
expert groups, countries ensure that the instruments are internationally valid and take into account the cultural 
and educational contexts in OECD member countries. They also ensure that the assessment materials have 
strong measurement properties and that the instruments emphasise authenticity and educational validity. 

Participating countries implement PISA at the national level, through National Project Managers (NPM), subject to 
the agreed administration procedures. National Project Managers play a vital role in ensuring that implementation 
is of high quality. They also verify and evaluate the survey results, analyses, reports and publications. 

The design of the assessment of reading, mathematics and science, and the implementation of the present 
survey, within the framework established by the PGB, is the responsibility of an international consortium led 
by the Australian Council for Educational Research (ACER). Other partners in this consortium include cApStAn 
Linguistic Quality Control and the Department of Experimental and Theoretical Pedagogy at the University of 
Liege (SPe) in Belgium, the Deutsches Institut fuer Padagogische Forschung (DIPF) in Germany, the National 
Institute for Educational Policy Research (NIER) in Japan, and WESTAT in the United States. 

The questionnaire development of the survey is carried out by a consortium led by the CITO Institute for 
Educational Measurement in the Netherlands. Other partners in this consortium include the Institute for 
Educational Research at the University ofjyvaskyla in Finland, the Direction de I'Evaluation de la Prospective et 
de la Performance (DEPP) in France, and the University of Twente in the Netherlands. The OECD Secretariat has 
overall managerial responsibility for the programme, monitors its implementation on a day-to-day basis, acts as 
the secretariat for the PGB, builds consensus among countries and serves as the interlocutor between the PGB 
and the international consortium charged with implementation. The OECD Secretariat is also responsible for 
the production of the indicators, and the analysis and preparation of the international reports and publications 
in co-operation with the PISA consortium, in close consultation with member countries both at the policy level 
(PGB) and at the implementation level (National Project Managers). 

The development of the PISA frameworks has been a continuous effort since the programme was created in 
1 997 and can be described as a sequence: 

■ Development of a working definition for the assessment domain and description of the assumptions that 
underlie that definition. 
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■ Evaluation of how to organise the tasks constructed in order to report to policy makers and researchers 
on student achievement in the domain, and identification of key characteristics that should be taken into 
account when constructing assessment tasks for international use. 

■ Operationalisation of key characteristics used in test construction, with definitions based on existing literature 
and experience in conducting other large-scale assessments. 

■ Validation of the variables and assessment of the contribution they each make to understanding task difficulty 
across the participating countries. 

■ Preparation of an interpretative scheme for the results. 

While the main benefit of constructing and validating a framework for each of the domains is improved 

measurement, there are other potential benefits: 

■ A framework provides a common language and a vehicle for discussing the purpose of the assessment and 
what it is trying to measure. Such a discussion encourages the development of a consensus around the 
framework and the measurement goals. 

■ An analysis of the kinds of knowledge and skills associated with successful performance provides a basis for 
establishing standards or levels of proficiency. As the understanding of what is being measured and the ability 
to interpret scores along a particular scale evolve, an empirical basis for communicating a richer body of 
information to various constituencies can be developed. 

■ Identifying and understanding particular variables that underlie successful performance further the ability to 
evaluate what is being measured and to make changes to the assessment over time. 

■ The understanding of what is being measured and its connection to what we say about students provides an 
important link between public policy, assessment and research which, in turn, enhances the usefulness of 
the data collected. 
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This chapter discusses the conceptual framework underlying 
the PISA 2009 assessment of students' reading competencies. 
It provides PISA's definition of reading literacy and presents 
the elements of the survey which have remained consistent 
throughout the previous cycles, along with a new element: 
reading and understanding electronic texts. It describes how 
PISA assesses and analyses electronic reading tasks, as well as 
the way in which students navigate through texts and respond to 
the format of tasks. Sample print and electronic reading items are 
included throughout the chapter to further illustrate how students' 
skill are measured. Finally, a discussion on reading engagement 
and metacognition addresses the motivational and behavioural 
elements of reading literacy. 
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INTRODUCTION 

Continuity and change in the reading literacy framework 

Reading literacy was the major domain assessed in 2000 for the first PISA cycle (PISA 2000). For the fourth 
PISA cycle (PISA 2009), it is the first of the domains to be revisited as a major focus, requiring a full review of 
its framework and new development of the instruments that represent it. 

The original reading literacy framework for PISA was developed for the PISA 2000 cycle (from 1998 to 2001) 
through a consensus building process involving reading experts selected by the participating countries and the PISA 
2000 advisory groups. The definition of reading literacy evolved in part from the lEA Reading Literacy Study (1 992) 
and the International Adult Literacy Survey (lALS, 1994, 1997 and 1998). In particular, it reflected lALS' emphasis 
on the importance of reading skills for active participation in society. It was also influenced by contemporary - 
and still current -theories of reading, which emphasise reading's interactive nature (Dechant, 1991; McCormick, 
1988; Rumelhart, 1985), models of discourse comprehension (Graesser, Millis, & Zwaan, 1997; Kintsch, 1998), 
and theories of performance in solving reading tasks (Kirsch, 2001; Kirsch & Mosenthal, 1990). 

Much of the substance of the PISA 2000 framework is retained in the PISA 2009 framework, respecting one of the 
central purposes of PISA: to collect and report trend information about performance in reading, mathematics and 
science. However, the PISA domain frameworks also aim to be evolving documents that will adapt to and integrate 
new developments in theory and practice over time. There is therefore a significant amount of evolution, reflecting 
both an expansion in our understanding of the nature of reading and changes in the world. 

There are two major modifications in this new version of the reading framework. It incorporates the reading of 
electronic texts and elaborates the constructs of reading engagement and metacognition. 

The PISA 2000 reading literacy framework briefly mentioned electronic texts, stating, "It is expected that 
electronic texts will be used in future survey cycles but will not be included in this cycle because of time and 
access issues (OECD, 1999). The PISA 2009 cycle is now upon us, and with it, recognition of the increasing 
prevalence of digital texts in many parts of our lives: personal, social and economic. The new demands on 
reading proficiency created by the digital world have led to the framework's inclusion of electronic reading, an 
inclusion that has in turn resulted in some redefinition both of texts and of the mental processes that readers use 
to approach texts. This edition of the framework thereby acknowledges the fact that any definition of reading in 
the 2D' century needs to encompass both printed and digital texts. 

PISA is the first large-scale international study to assess electronic reading. As such, this initiative, while 
grounded in current theory and best practices from around the world, is inevitably a first step. This reality is 
reflected in the fact that not all participating countries have elected to take part in the administration of the 
electronic reading assessment in PISA 2009, which has therefore been implemented as an international option. 
The assessment of electronic reading will be reviewed and refined over successive cycles to keep pace with 
developing technologies, assessment tools and conceptual understanding of the electronic medium's impact. 

Changes in our concept of reading since 2000 have already led to an expanded definition of reading literacy, 
which recognises motivational and behavioural characteristics of reading alongside cognitive characteristics. 
Both reading engagement and metacognition - an awareness and understanding of how one thinks and uses 
thinking strategies - were referred to briefly at the end of the first PISA framework for reading under "Other issues" 
(OECD, 1999). In the light of recent research, reading engagement and metacognition are featured more 
prominently in this PISA 2009 reading framework as elements that can make an important contribution to policy 
makers' understanding of factors that can be developed, shaped and fostered as components of reading literacy. 



THE STRUCTURE OF THE READING LITERACY FRAMEWORK 

This chapter addresses what is meant by the term reading literacy in PISA, and how it will be measured in 
PISA 2009. This section introduces the importance of reading literacy in today's societies. The second section 
defines reading literacy and elaborates on various phrases that are used in the reading framework, along with 
the assumptions underlying the use of these words. The third section focuses on the organisation of the domain 
of the assessment of reading literacy, and discusses the characteristics that will be represented in the tasks 
included in the PISA 2009 assessment. The fourth section discusses some of the operational aspects of the 
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assessment. The fifth section describes the theoretical basis for the constructs of engagement and metacognition 
in the context of reading, and outlines approaches for measuring those constructs. Finally, the last section 
describes how the reading literacy data will be summarised and outlines plans for reporting. 

Reading literacy as a foundational skill 

We live in a rapidly changing world, where both the quantity and type of written materials are increasing and 
where more and more people are expected to use these materials in new and sometimes more complex ways. 
It is now generally accepted that our understanding of reading literacy evolves along with changes in society 
and culture. The reading literacy skills needed for individual growth, economic participation and citizenship 
20 years ago were different from those of today; and it is likely that in 20 years time they will change further still. 

The goal of education has shifted its emphasis from the collection and memorisation of information only, 
to the inclusion of a broader concept of knowledge: The meaning of knowing has shifted from being able 
to remember information, to being able to find and use it" (Simon, 1996). The ability to access, understand 
and reflect on all kinds of information is essential if individuals are to be able to participate fully in our 
knowledge-based society. The PISA framework for assessing the reading literacy of students towards the end 
of compulsory education, therefore, must focus on reading literacy skills that include finding, selecting, 
interpreting and evaluating information from the full range of texts associated with situations that reach 
beyond the classroom. 

According to Holloway (1999), reading skills are essential to the academic achievement of middle- and high- 
school students. Olson (1977a; 1977b) claims that in today's society, reading literacy introduces a bias because 
it provides advantages to those who acquire the necessary skills. As the currency used in schools, literacy 
provides access to literate institutions and has an impact on cognition, or thinking processes (Olson, 1994); 
it also shapes the way in which we think. 

Achievement in reading literacy is not only a foundation for achievement in other subject areas within the 
educational system, but also a prerequisite for successful participation in most areas of adult life (Cunningham 
& Stanovich, 1998; Smith, Mikulecky, Kibby, & Dreher, 2000). 

Today, the need for higher levels of education and skills is large and growing. Those with below average skills 
find it increasingly difficult to earn above average wages in global economies where the restructuring of jobs 
favours those who have acquired higher levels of education and skills. They have little hope of fully participating 
in increasingly complex societies where individuals are required to take on additional responsibility for different 
aspects of their lives: from planning their careers, to nurturing and guiding their children, to navigating health- 
care systems, to assuming more responsibility for their financial future. The non-economic returns from literacy 
in the form of enhanced personal well-being and greater social cohesion are as important as the economic 
and labour-market returns, according to some authorities (Friedman, 2005; OECD, 2001). Elwert (2001) has 
advanced the concept of societal literacy , referring to the way in which literacy is fundamental in dealing 
with the institutions of a modern bureaucratic society. Law, commerce and science use written documents 
and written procedures such as laws, contracts and publications that one has to be able to understand in 
order to function in these domains. The European Commission (2001) summed up the foundational nature 
of reading literacy skills as key to all areas of education and beyond, facilitating participation in the wider 
context of lifelong learning and contributing to individuals' social integration and personal development". More 
recently, the European Union endorsed this statement with its enshrinement of communication in the mother 
tongue, comprising listening, speaking, reading and writing, as the first of eight key compentencies "which all 
individuals need for personal fulfilment and development, active citizenship, social inclusion and employment" 
(Education Council, 2006). 

Reading literacy skills matter not just for individuals, but for economies as a whole. Policy makers and others are 
coming to recognise that in modern societies, human capital - the sum of what the individuals in an economy 
know and can do - may be the most important form of capital. Economists have for many years developed 
models showing generally that a country's education levels are a predictor of its economic growth potential. 
Although the strength of this link is limited by the fact that an educational credential means something different 
from one country to another, international surveys such as the International Adult Literacy Survey (lALS) or 
the upcoming OECD Programme for the International Assessment of Adult Competencies (PIAAC) now let us 
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measure adults' literacy skills directly and not just through their credentials. These surveys, in turn, allow us to 
make more credible inferences about the connection between human capital and national economic growth. In 
a recent study, several Canadian economists analysed links between literacy levels and economic performance 
over a long period. They found that the average literacy level of a nation's population is a better predictor of 
economic growth than educational achievement (Coulombe, Trembly, & Marchand, 2004). 

The importance of electronic texts 

Proficiency in reading literacy is a key not only to unlocking the world of printed text, but also electronic 
texts, which are becoming an increasingly important part of students' and adults' reading. As of 2007, almost 
1 .5 billion people -one-fifth of the world's population - were reading on line (International Telecommunications 
Union, 2009). The rate of growth in online use has been staggering, with much of it having occurred during the 
past five years - though the rate varies widely according to location (The World Bank, 2007). The variation is 
not only geographical, but also social and economic. In all countries, Internet use is closely linked with socio- 
economic status and education (Sweets & Meates, 2004). Yet the requirement to use computers is not confined 
to particular social and economic strata. The Adult Literacy and Life Skills Survey (OECD and STATCAN, 2005) 
looked at computer use by type of occupation in seven countries or regions. While "expert" knowledge workers 
such as scientists and computing professionals use computers most intensively in the workplace, office workers 
and customer service clerks are also likely to need to use computers on the job. Therefore workers in a wide 
range of occupations are increasingly required to use computers as part of their jobs. 

Beyond the workplace, computer technology has a growing importance in personal, social and civic life. To 
stay informed and involved, accessing information via networked computer technologies is becoming the norm. 
As individuals take on more responsibility for health, retirement and finance decisions, these technologies 
become increasingly important sources of information. Those with access to the Internet and with the skills and 
knowledge to use it effectively are more likely to become empowered patients who can make informed health- 
care choices; active citizens who use e-mail to influence government officials' policy decisions or mobilise 
like-minded voters; and members of virtual communities who, via online support groups, use instant messaging 
and discussion boards to interact with others across social classes, racial groups and generations (Pew Internet 
& American Life Project, 2005). 

While many of the skills required for print and electronic reading are similar, electronic reading demands that 
new emphases and strategies be added to the repertoires of readers. Gathering information on the Internet 
requires skimming and scanning through large amounts of material and immediately evaluating its credibility. 
Critical thinking, therefore, has become more important than ever in reading literacy (Halpern, 1989; Shetzer 
& Warschauer, 2000; Warschauer, 1999). Warschauer concludes that overcoming the "digital divide" is not 
only a matter of achieving online access, but also of enhancing people's abilities to integrate, evaluate and 
communicate information. 

Motivational and behavioural elements of reading literacy 

Reading-related skills, attitudes, interests, habits and behaviours have been shown in a number of recent studies 
to be strongly linked with reading proficiency. For example, in PISA 2000 there was a greater correlation 
between reading proficiency and reading engagement (comprising attitudes, interests and practices) than 
between reading proficiency and socio-economic status (OECD, 2002). In other studies reading engagement 
has been shown to account for more variance in reading achievement than any other variable besides previous 
achievement (Guthrie & Wigfield, 2000). 

Like reading engagement, metacognition has long been considered to be related to reading achievement 
(Brown, Brown, etal. 1983; Elavell & Wellman, 1977; Schneider, 1989, 1999; Schneider & Pressley, 1997), but 
most studies of metacognition have been largely experimental and focused on young readers. The PISA 2000 
reading framework alluded to the potential for using PISA to collect information about metacognition relevant 
to policy makers, but concluded that in the absence of an existing instrument suitable for use in a large-scale 
study, metacognition could not be part of the reading literacy study in 2000 (OECD, 1999). Since then, such 
instrumentation has been developed (Artelt, Schiefele, & Schneider, 2001; Schlagmilller & Schneider, 2006) 
making the inclusion of a survey of metacognition in reading within PISA 2009 feasible. 
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There is evidence that skills relating to engagement and metacognition can be taught. Interest in measuring both 
metacognition and engagement as part of PISA 2009 therefore assumes that results can yield information that 
will be highly relevant to policy makers and that can also influence the practice of reading and learning and 
ultimately levels of reading proficiency. 



DEFINING READING LITERACY 

Definitions of reading and reading literacy have changed over time in parallel with changes in society, economy, 
and culture. The concept of learning, and particularly the concept of lifelong learning, have expanded the 
perception of reading literacy. Literacy is no longer considered an ability acquired only in childhood during the 
early years of schooling. Instead it is viewed as an expanding set of knowledge, skills and strategies that individuals 
build on throughout life in various contexts, through interaction with their peers and the wider community. 

Cognitively-based theories of reading literacy emphasise the interactive nature of reading and the constructive 
nature of comprehension, in the print medium (Binkley & Linnakyla, 1997; Bruner, 1990; Dole, Duffy, Roehler, 
& Pearson, 1991) and to an even greater extent in the electronic medium (Fastrez, 2001; Legros & Crinon, 2002; 
Leu, 2007; Reinking, 1994). The reader generates meaning in response to text by using previous knowledge and 
a range of text and situational cues that are often socially and culturally derived. While constructing meaning, the 
reader uses various processes, skills, and strategies to foster, monitor, and maintain understanding. These processes 
and strategies are expected to vary with context and purpose as readers interact with a variety of continuous and 
non-continuous texts in the print medium and (typically) with multiple texts in the electronic medium. 

The PISA 2000 definition of reading literacy is as follows: 

Reading literacy is understanding, using and reflecting on written texts, in order to achieve one's goals, 
to develop one's knowledge and potential, and to participate in society. 

The PISA 2009 definition of reading adds engagement in reading as an integral part of reading literacy: 

Reading literacy is understanding, using, reflecting on and engaging with written texts, in order to achieve 
one's goals, to develop one's knowledge and potential, and to participate in society. 

Each part of the definition is considered in turn below, taking into account the original elaboration and some 
important developments in the defining of the domain which use evidence from PISA and other empirical 
studies, from theoretical advances and from the changing nature of the world. 

Reading literacy . . . 

The term "reading literacy" is preferred to "reading" because it is likely to convey to a non-expert audience more 
precisely what the survey is measuring. "Reading" is often understood as simply decoding, or even reading aloud, 
whereas the intention of this survey is to measure something broader and deeper. Reading literacy includes a wide 
range of cognitive competencies, from basic decoding, to knowledge of words, grammar and larger linguistic and 
textual structures and features, to knowledge about the world. It also includes metacognitive competencies: the 
awareness of and ability to use a variety of appropriate strategies when processing texts. Metacognitive competencies 
are activated when readers think about, monitor and adjust their reading activity for a particular goal. 

Historically, the term literacy referred to a tool used to acquire and communicate writtten and printed information. 
This seems close to the notion that the term "reading literacy" is intended to express in this study: the active, purposeful 
and functional application of reading in a range of situations and for various purposes. PISA assesses a wide range 
of students. Some of these students will go on to a university, possibly to pursue an academic career; some will 
pursue further studies in preparation for joining the labour force; and some will enter the workforce directly upon 
completion of school education. Regardless of their academic or labour-force aspirations, reading literacy will be 
important to their active participation in their community and economic and personal life. 

. . is understanding, using, reflecting on .. . 

The word understanding is readily connected with reading comprehension , a well-accepted element of 
reading. The word "using" refers to the notions of application and function - doing something with what we 
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read. "Reflecting on" is added to "understanding" and "using" to emphasise the notion that reading is interactive: 
readers draw on their own thoughts and experiences when engaging with a text. Of course, every act of reading 
requires some reflection, drawing on information from outside the text. Even at the earliest stages, readers draw 
on symbolic knowledge to decode a text and require a knowledge of vocabulary to make meaning. As readers 
develop their stores of information, experience and beliefs, they constantly, often unconsciously, test what they 
read against outside knowledge, thereby continually reviewing and revising their sense of the text. At the same 
time, incrementally and perhaps imperceptibly, readers' reflections on texts may alter their sense of the world. 
Reflection might also require readers to consider the content of the text, apply their previous knowledge or 
understanding, or think about the structure or form of the text. 

As it is not possible to include sufficient items from the PISA assessment to report on each of the five aspects 
as a separate subscale, for reporting on reading literacy, these five aspects are organised into three broad 
aspect categories. In PISA 2000, PISA 2003 and PISA 2006 these three broad aspects were called "Retrieving 
information", "Interpreting texts" and "Reflecting and evaluating" respectively. The terms have been changed 
for PISA 2009 to better accommodate the aspects in relation to electronic texts. 

. . . and engaging with . . . 

A reading literate person not only has the skills and knowledge to read well, but also values and uses reading 
for a variety of purposes. It is therefore a goal of education to cultivate not only proficiency but also engagement 
in reading. Engagement in this context implies the motivation to read and is comprised of a cluster of affective 
and behavioural characteristics that include an interest in and enjoyment of reading, a sense of control over 
what one reads, involvement in the social dimension of reading, and diverse and frequent reading practices. 

. . . written texts . . . 

The phrase "written texts" is meant to include all those coherent texts in which language is used in its graphic 
form: hand-written, printed and electronic. These texts do not include aural language artefacts such as voice 
recordings; nor do they include film, TV, animated visuals, or pictures without words. They do include visual 
displays such as diagrams, pictures, maps, tables, graphs and comic strips, which include some written 
language (for example, captions). These visual texts can exist either independently or they can be embedded 
in larger texts. "Hand-written texts" are mentioned for completeness: although they are clearly part of the 
universe of written texts, they are not very different from printed texts in structure or in terms of the processes 
and reading strategies they require. Electronic texts, on the other hand, are distinguished from printed texts in 
a number of respects, including physical readability; the amount of text visible to the reader at any one time; 
the way different parts of a text and different texts are connected with one another through hypertext links; and 
consequent upon all these text characteristics, the way that readers typically engage with electronic texts. To 
a much greater extent than with printed or hand-written texts readers need to construct their own pathways to 
complete any reading activity associated with an electronic text. 

Instead of the word "information", which is used in some other definitions of reading, the term "texts" was 
chosen because of its association with written language and because it more readily connotes literary as well 
as information-focused reading. 

... in order to achieve one's goals, to develop one's knowledge and potential, and to participate 
in society. 

This phrase is meant to capture the full scope of situations in which reading literacy plays a role, from private 
to public, from school to work, from formal education to lifelong learning and active citizenship. To achieve 
one's goals and to develop one's knowledge and potential" spells out the idea that reading literacy enables the 
fulfilment of individual aspirations - both defined ones such as graduatingor getting a job, and those less defined 
and less immediate which enrich and extend personal life and lifelong education. The word "participate" is 
used because it implies that reading literacy allows people to contribute to society as well as to meet their own 
needs: "participating" includes social, cultural, and political engagement. Literate people, for example, find it 
easier to navigate complex institutions such as health systems, government offices and legal agencies; and they 
can participate more fully in a democratic society by making informed decisions when they vote. Participation 
may also include a critical stance, a step for personal liberation, emancipation, and empowerment (Linnakyla, 
1992; Lundberg, 1991, 1997; MacCarthey & Raphael, 1989). 
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Fifty years ago in his seminal work Maturity in Reading Gray wrote of the interests, attitudes and skills 
that enable young people and adults to meet effectively the reading demands of their current lives (Gray 
& Rogers, 1956). The PISA concept of reading literacy is consistent with Gray's broad and deep notion of 
maturity in reading, while simultaneously embracing the new challenges of reading in the 21®' century. It 
conceives reading as the foundation for full participation in the economic, political, communal and cultural 
life of contemporary society. 



ORGANISING THE DOMAIN 

The previous section defined the domain of reading literacy and laid out the set of assumptions that were made 
in constructing this definition. This section describes how the domain is represented, a vital issue because the 
organisation and representation of the domain determines the test design and, ultimately, the evidence about 
student proficiencies that can be collected and reported.' 

Reading is a multidimensional domain. While many elements are part of the construct, not all can be taken into 
account and manipulated in an assessment such as PISA. In designing an assessment it is necessary to select the 
elements considered most important to manipulate in building the assessment. 

For PISA, the two most important considerations are firstly, to ensure broad coverage of what students read and 
for what purposes they read, both in and outside of school; and secondly, to organise the domain to represent 
a range of difficulty. The PISA reading literacy assessment is built on three major task characteristics: situation - 
the range of broad contexts or purposes for which reading takes place; text - the range of material that is read; 
and aspect - the cognitive approach that determines how readers engage with a text. All three contribute to 
ensuring broad coverage of the domain. In PISA, features of the text and aspect variables (but not of the situation 
variable) are also manipulated to influence the difficulty of a task. 

In order to use these three main task characteristics in designing the assessment and, later, interpreting the 
results, they must be operationalised. That is, the various values that each of these characteristics can take on 
must be specified. This allows test developers to categorise the materials they are working with and the tasks 
they construct so that they can then be used to organise the reporting of the data and to interpret results. 

Reading is a complex activity; the components of reading therefore do not exist independently of one another in 
neat compartments. The assignment of texts and tasks to framework categories does not imply that the categories 
are strictly partitioned or that the materials exist in atomised cells determined by a theoretical structure. The 
framework scheme is provided to ensure coverage, to guide the development of the assessment and to set 
parameters for reporting, based on what are considered the marked features of each task. 

Situation 

A useful operationalisation of the situation variables is found in the Common European Framework of Reference 
(CEFR) developed for the Council of Europe (Council of Europe, 1 996). Although this framework was originally 
intended to describe second- and foreign- language learning, in this respect at least it is relevant to mother- 
tongue language assessment as well. The CEFR situation categories are: reading for private use; reading for 
public use; reading for work and reading for education. They have been adapted for PISA to personal, public, 
occupational and educational contexts, and are described in the paragraphs below. 

The persona/ category relates to texts that are intended to satisfy an individual's personal interests, both practical 
and intellectual. This category also includes texts that are intended to maintain or develop personal connections 
with other people. It includes personal letters, fiction, biography, and informational texts that are intended to 
be read to satisfy curiosity, as a part of leisure or recreational activities. In the electronic medium it includes 
personal e-mails, instant messages and diary-style blogs. 

The public category describes the reading of texts that relate to activities and concerns of the larger society. The 
category includes official documents as well as information about public events. In general, the texts associated 
with this category assume a more or less anonymous contact with others; they also therefore include forum- 
style blogs, news websites and public notices that are encountered both on line and in print. 
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The content of educational texts is usually designed specifically for the purpose of instruction. Printed text 
books and interactive learning software are typical examples of material generated for this kind of reading. 
Educational reading normally involves acquiring information as part of a larger learning task. The materials 
are often not chosen by the reader, but instead assigned by an instructor. The model tasks are those usually 
identified as "reading to learn" (Sticht, 1975; Stiggins, 1982). 



Many 15-year-olds will move from school into the labour force within one to two years. A typical occupational 
reading task is one that involves the accomplishment of some immediate task. It might include searching for a job, 
either in a print newspaper's classified advertisement section, or on line; or following workplace directions. The 
model tasks of this type are often referred to as "reading to do" (Sticht, 1 975; Stiggins, 1 982). Texts written for these 
purposes, and the tasks based on them, are classified as occupational in PISA. While only some of the 1 5-year-olds 
who are assessed will currently have to read at work, it is important to include tasks based on texts that are related to 
work since the assessment of young people's readiness for life beyond compulsory schooling and their ability to use 
their knowledge and skills to meet real-life challenges is a fundamental goal of PISA. 



Situation is used in PISA reading literacy to define texts and their associated tasks, and refers to the contexts 
and uses for which the author constructed the text. The manner in which the situation variable is specified is 
therefore about supposed audience and purpose, and is not simply based on the place where the reading activity 
is carried out. Many texts used in classrooms are not specifically designed for classroom use. For example, a 
piece of literary text may typically be read by a 1 5-year-old in a mother-tongue language or literature class, yet 
the text was written (presumably) for readers' personal enjoyment and appreciation. Given its original purpose, 
such a text is classified as personal in PISA. As Hubbard (1989) has shown, some kinds of reading usually 
associated with out-of-school settings for children, such as rules for clubs and records of games, often take place 
unofficially at school as well. These texts are classified as public in PISA. Conversely, textbooks are read both 
in schools and in homes, and the process and purpose probably differ little from one setting to another. Such 
texts are classified as educational in PISA. 



It should be noted that the four categories overlap. In practice, for example, a text may be intended both to 
delight and to instruct (personal and educational); or to provide professional advice which is also general 
information (occupational and public). While content is not a variable that is specifically manipulated in this 
study, by sampling texts across a variety of situations the intent is to maximise the diversity of content that will 
be included in the PISA reading literacy survey. 

One obvious way to distribute the reading literacy tasks in the assessment would be to do so evenly across 
the four situations. In the PISA 2000 framework however the occupational situation is given less weight for 
two reasons. First, it was considered important to reduce the potential dependence on specific occupational 
knowledge that can result when selecting occupational texts. Second, it was expected that the same type 
of questions and directives could be constructed from texts classified in one of the other situations, where 
1 5-year-old students might have better access to the content. These considerations remain relevant in 2009. The 
distribution of tasks by situation for PISA 2009 print reading is therefore very similar to that for 2000. Table 1 .1 
shows the approximate distribution of tasks by situation for print and electronic reading tasks. It should be noted 
that the percentages given here and in all other tables in this section are approximate only, as distribution of 
tasks according to framework variables is not final at the time of publication. 



Tgblel 


1 Approximate distribution of tasks by situation for PISA 2009 


Situation 


% of total tasks PISA 2009: print 


% of total tasks PISA 2009: electronic 


Personal 


30 


30 


Educational 


25 


15 


Occupational 


15 


15 


Public 


30 


40 


TOTAL 


100 


100 
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Text 

Reading requires material for the reader to read. In an assessment, that material - a text (or a set of texts) 
related to a particular task - must be coherent within itself. That is, the text must be able to stand alone without 
requiring additional material to make sense to the proficient reader.^ While it is obvious that there are many 
different kinds of texts and that any assessment should include a broad range. It Is not so obvious that there is 
an ideal categorisation of kinds of texts. The addition of electronic reading in the 2009 framework makes this 
issue still more complex. For PISA 2009 there are four main classifications: 

1 . Medium: print and electronic 

2. Environment: authored and message-based 

3. Text format: continuous, non-continuous, mixed and multiple 

4. Text type: description, narration, exposition, argumentation, instruction and transaction 

The classification of medium - print and electronic - is applied to each text as the broadest distinction. Below 
that classification, the text format and text type categories are applied to all texts, whether print or electronic. 
The environment classification, on the other hand, is only applicable to electronic-medium texts. Each of these 
characteristics is discussed below. 

In addition to the four major text characteristics - medium, environment, text format and text type - some 
additional terms are used in the following sections to describe characteristics of both print and electronic texts. 

Text object is a term used to describe the familiar names given to texts when we refer to them in everyday 
contexts: terms such as report, novel, play, timetable, home page or e-mail message. Text objects vary according 
to both medium and text format. For example, timetables occur as non-continuous texts in both print and 
electronic media; home pages occur only in the electronic medium; reports may appear in either medium and 
in a variety of text formats. 

Text features are characteristics of the text-based information that students have to work with in a task. Text 
features include the number of texts or pages students need to read in order to respond to individual items, the 
length of the texts to be read, the linguistic complexity of the texts, and the assumed familiarity the students 
have with the topics presented. 

Navigation tools and features help readers to negotiate their way into, around and across texts. Navigation tools 
and features are discussed below in the context of electronic-medium texts. They include navigation icons, 
scroll bars, tabs, menus, embedded hyperlinks, text search functions such as Find or Search, and global content 
representation devices such as site maps. Many navigation tools and features are intrinsic and unique to the 
electronic medium, and make up some of its defining characteristics. However like many of the other electronic 
text elements, navigation tools and features have parallels in the print medium. In print they include tables of 
contents, indexes, chapter and section headings, headers and footers, page numbers and footnotes. 

Medium 

An important major categorisation of texts, new in the PISA 2009 framework for reading literacy, is the 
classification by medium: print or electronic. 

Print-medium text usually appears on paper in forms such as single sheets, brochures, magazines and books. 
The physical status of the printed text encourages (though it may not compel) the reader to approach the content 
of the text in a particular sequence. In essence, printed texts have a fixed or static existence. Moreover, in real 
life and in the assessment context, the extent or amount of the text is immediately visible to the reader. 

Electronic-medium text may be defined as the display of text through Liquid Crystal Display (LCD), plasma. 
Thin Film Transistor (TFT) and other electronic devices. For the purposes of PISA, however, electronic text is 
synonymous with hypertext: a text or texts with navigation tools and features that make possible and indeed even 
require non-sequential reading. Each reader constructs a "customised" text from the information encountered 
at the links he or she follows. In essence, such electronic texts have an unfixed, dynamic existence. In the 
electronic medium, typically only a fraction of the available text can be seen at any one time, and often the 
extent of text available is unknown. 
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Fi^ufe1.1 ■ Print reading texts in PIS A 



Fixed text with 
defined boundaries 



Fi^ute 1.2" Electronic reading texts in PISA 



Navigation tools 
and features 



Dynamic text with 
blurred boundaries 



Multiple texts 



The difference between texts in the print and electronic media, in the PISA assessment context, is illustrated in 
Figure 1 .1 and Figure 1 .2. 

Navigation tools and features play a particularly important role in the electronic medium, for at least two reasons. 
Firstly, due to the reduced display size, electronic texts come with devices that let the reader move the reading 
window over the text page: scroll bars, buttons, index tabs and so forth. Skilled readers of electronic text must be 
familiar with the use of these devices. They must also be able to mentally represent the movement of the window 
over the text page, and the shifting from one window to another. Secondly, typical electronic reading activities 
involve the use of multiple texts, sometimes selecting from a virtually infinite pool. Readers must be familiar with the 
use of retrieval, indexing and navigation tools for linking between texts. 

One of the earliest indexing techniques used in electronic documents was the menu or list of page headings 
from which the reader is invited to choose. The electronic menu resembles a table of contents except that 
there are usually no page numbers. Instead, the reader selects an option by typing in its number in the menu 
or by clicking directly the menu option or a symbol that represents it. This results in the display of the pages 
instead of, or sometimes (in multi-window displays) on top of, the menu page. A consequence of the lack 
of page numbers is that once the page is displayed, the reader has no direct clue about its position among 
the set that makes up the electronic book. Sometimes such clues are provided through analogical symbols 
(for example, a micro-page within a series of micro-pages at the bottom of the screen) or through path-type 
expressions. Menus can be made hierarchical, which means that selecting a menu item causes another, more 
specific menu to be displayed. Menus may be presented as separate pages, but they may also be presented 
as part of multi-text pages. They are often presented in a frame to the left of the display window. The rest of 
the window can be updated with the menu remaining constant, which is considered helpful for the reader to 
keep a sense of his or her "location" in the document. Skilled reading of electronic texts therefore requires an 
understanding of hierarchical and straight-list menus, as well as an ability to mentally represent non-sequential 
page arrangements, whether hierarchical or networked. 

A major navigation tool that assists readers in finding their way around a number of texts, and one of the most 
distinctive features of electronic texts, is the hypertext link, a technique that appeared in the 1 980s as a means 
to connect units of information in large electronic documents (Conklin, 1987; Koved & Shneiderman, 1986; 
Lachman, 1989; Weyer, 1982). The hypertext link or hyperlink is a piece of information (a word or phrase, 
or a picture or icon) that is logically connected to another piece of information (usually a page). Clicking a 
hyperlink results in the display of a new page instead of or on top of the page previously displayed, or the 
display of another location on the same page. Hyperlinks may be presented in separate lists (also called menus) 
or embedded within content pages. When embedded, hyperlinks are generally marked using a specific colour 
or typography. The use of hyperlinks allows the creation of multi-page documents with a networked structure. 
Unlike lists or hierarchies, the arrangement of pages in a networked structure is not regularised according to a 
systematic set of conventions. Rather, it follows the semantic relationships across pages. It is up to the author of 
a multi-page electronic document to determine how the pages are linked, through the insertion of hyperlinks. 

Navigation and orientation within non-sequential structures seem to rely on the reader's ability to mentally 
represent the top-level structure of the hypertext. Global organisers that accurately represent the structure of 
pages and links (for example, structured menus and content maps) are usually of some help, provided that such 
organisers use symbols and metaphors that are already familiar to the reader (Rouet & Potelle, 2005). 
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Skilled reading, navigation and information search in hypertext requires the reader to be familiar with explicit 
and embedded hyperlinks, non-sequential page structures, and global content representation devices. 

In the PISA 2009 assessment of electronic reading (ERA), a set of navigation tools and structures have been 
identified for systematic inclusion in the instruments, as one important component in measuring proficiency 
in electronic reading. This set includes scroll bars for moving up and down a page; tabs for different websites; 
lists of hyperlinks displayed in a row, in a column or as a drop-down menu; and embedded hyperlinks - that 
is, hyperlinks included in paragraphs, tables of information or a list of search results. Hyperlinks may take the 
form of icons or words. 

The difficulty of a task is partly conditioned by the navigation tools and features associated with it. Tasks are 
more or less easy depending on the number of navigation tools that are required to be used, the number of 
operations or steps required, and the type of tools used. Generally, the larger the number of operations, and 
the more complex the tool type, the greater the item difficulty. The familiarity, transparency or prominence of 
navigation tools and features also affects difficulty. For example, a hyperlink labelled "click here" is typically 
easier to navigate than a drop down menu that only displays itself if the cursor passes over it. Some electronic 
reading tasks require little or even no navigation: for example, when students are required to locate or interpret 
information on a web page where the text is fully visible. 

Environment 

The environment classification is a new variable for the PISA 2009 reading framework. In PISA it applies only 
to electronic-medium texts. 

Electronic texts exist in a number of environments, including the web environment, desktop and e-mail. There 
are other electronic environments that use written text such as mobile phone text messages and electronic 
diaries. For the purposes of PISA 2009, however, only computer-based environments are considered. 

Two broad kinds of electronic environment have been identified for the assessment of reading of electronic 
texts. The distinction between them is based on whether or not the reader has the potential to influence the 
content of the site. An authored environment is one in which the reader is primarily receptive: the content 
cannot be modified. A message-based environment is one in which the reader has the opportunity to add to or 
change the content. 

Texts in an authored environment have a fixed content that cannot be influenced by the reader. They are self- 
contained environments, controlled or published by a commercial company, a government department, an 
organisation or institution, or an individual. Readers use these sites mainly for obtaining information. Text 
objects within an authored environment include home pages, sites publicising events or goods, government 
information sites, educational sites containing information for students, news sites and online library catalogues. 

In a message-based environment, on the other hand, the reader is invited to participate and contribute in some way. 
The content is to some extent fluid or collaborative, in that it can be added to or changed in some way by individuals. 
Readers use these sites not only for obtaining information, but also as a way of communicating. Text objects within a 
message-based environment include e-mail, blogs, chat rooms, web forums and reviews, and online forms. 

Inevitably, the possible range of text object categories within each of the electronic environments is not 
completely represented given the limited number of tasks in the PISA 2009 instrument. Instead the assessment 
comprises a sample of the text objects likely to be encountered by 1 5-year-olds and young adults in educational, 
occupational, personal, and public contexts. 

As with many of the variables in the reading framework, the environment classifications are not strictly 
partitioned. A given website, for example, may include some authored text and a section in which the reader 
is invited to add a comment. Nevertheless, an individual task generally draws predominantly upon either an 
authored or a message-based part of the stimulus, and is classified acccordingly. Occasionally a task may 
require integrated use of both authored and message-based texts. Such tasks are classified as mixed. Table 1 .2 
shows the proportion of tasks in each environment category. 
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Tgble 1.2 Approximate distribution of electronic tasks by environment 





% tasks in electronic reading 


Environment 


assessment 


Authored 


70 


Message-based 


25 


Mixed 


5 


TOTAL 


100 



E 



The authored and message-based environments are not considered hierarchical in difficulty; nor are they 
intended to form the basis for reporting subscales. The distribution of tasks across the two broad environments 
has been used primarily to guide test development so that a range of reading situations is covered that is 
relevant to 15-year-olds both in and beyond school. 



Text format 

An important classification of texts, and one at the heart of the organisation of the PISA 2000 framework and 
assessment, is the distinction between continuous and non-continuous texts. Continuous texts are typically 
composed of sentences that are, in turn, organised into paragraphs. These may fit into even larger structures 
such as sections, chapters, and books. Non-continuous texts are most frequently organised in matrix format, 
based on combinations of lists. 

Texts in continuous and non-continuous format appear in both the print and electronic media. Mixed and 
multiple format texts are also prevalent in both media, particularly so in the electronic medium. Each of these 
four formats is elaborated below. 

Other non-text formatted objects are also commonly used in conjunction with print and particularly with 
electronic texts. Pictures and graphic images occur frequently in print texts and can legitimately be regarded as 
integral to such texts. Static images as well as videos, animations and audio files regularly accompany electronic 
texts and can, also, be regarded as integral to those texts. As a reading literacy assessment, PISA does not focus 
on non-text formatted objects independently, but any such objects may, in principle, appear in PISA as part of 
a (verbal) text. However in practice the use of video and animation is very limited in the current assessment. 
Audio is not used at all because of practical limitations such as the need for headphones and audio translation. 

Continuous texts 

Continuous texts are formed by sentences organised into paragraphs. 

Graphically or visually, organisation occurs by the separation of parts of the text into paragraphs, by paragraph 
indentation, and by the breakdown of text into a hierarchy signalled by headings that help readers to recognise 
the organisation of the text. These markers also provide clues to text boundaries (showing section completion, 
for example). The location of information is often facilitated by the use of different font sizes, font types such as 
italic and boldface, or borders and patterns. The use of format clues is an essential subskill of effective reading. 

Discourse markers also provide organisational information. Sequence markers (first, second, third, etc.), for 
example, signal the relation of each of the units introduced to each other and indicate how the units relate to the 
larger surrounding text. Causal connectors (therefore, for this reason, since, etc.) signify cause-effect relationships 
between parts of a text. 

Examples of text objects in continuous text format in the print medium include newspaper reports, essays, 
novels, short stories, reviews and letters. In the electronic medium the continuous text format group includes 
reviews, blogs and reports in prose. Electronic continuous texts tend to be short because of the limitations of 
screen size and the need for piecemeal reading, which make long texts unattractive to many online readers. 
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Non-continuous texts 

Non-continuous texts, also known as documents, are organised differently to continuous texts, and therefore 
require a different kind of reading approach. As the sentence is the smallest unit of continuous text, so all non- 
continuous texts can be shown to be composed of a number of lists (Kirsch & Mosenthal, 1990). Some are single, 
simple lists, but most consist of several simple lists combined. This analysis of non-continuous texts does not refer 
to their use or employ the common labels often attached to them, but does identify key structural features that are 
common to a number of different texts. Readers who understand the structure of texts are better able to identify the 
relationships between the elements and understand which texts are similar and which are different. 




Examples of non-continuous text objects are lists, tables, graphs, diagrams, advertisements, schedules, 
catalogues, indexes and forms. These text objects occur in both print and electronic media. 



The following two text format categories are new in the 2009 framework. Recognition of the importance of 
integrating information in different formats and across several texts, as part of the reader's repertoire, has led to 
the identification of mixed and multiple texts as distinct text formats. 



Mixed texts 

Many texts in both print and electronic media are single, coherent objects consisting of a set of elements in 
both a continuous and non-continuous format. In well-constructed mixed texts the components (for example, 
a prose explanation including a graph or table) are mutually supportive through coherence and cohesion links 
at the local and global level. 

Mixed text in the print medium is a common format in magazines, reference books and reports, where authors 
employ a variety of presentations to communicate information. In the electronic medium authored web pages 
are typically mixed texts, with combinations of lists, paragraphs of prose and often graphics. Message-based 
texts such as online forms, e-mail messages and forums also combine texts that are continuous and non- 
continuous in format. 



Multiple texts 

For the purposes of the PISA reading framework multiple texts are defined as those which have been generated 
independently, and make sense independently; they are juxtaposed for a particular occasion or may be loosely 
linked together for the purposes of the assessment. The relationship between the texts may not be obvious; they 
may be complementary or may contradict one another. For example, a set of websites from different companies 
providing travel advice may or may not provide similar directions to tourists. Multiple texts may have a single 
"pure" format (for example, continuous), or may include both continuous and non-continuous texts. 

Tasks in the print-medium assessment continue to be classified for the most part as either continuous or non- 
continuous, with about two-thirds of such tasks addressing continuous texts and one-third non-continuous 
texts. Although some mixed and multiple texts were used in the PISA 2000 assessment, they were not separately 
classified, but rather described in terms of their continuous or non-continuous elements. In the development 
of tasks for the PISA 2009 assessment there has been a more deliberate effort to include stimuli of mixed and 
multiple print texts, and to include tasks that require the reader to integrate information across differently 
formatted parts within a mixed text or across multiple texts. In previous administrations of PISA the few tasks 
that required integration within mixed texts or across multiple texts were classified according to text format on 
the basis of what was judged to be the part of the stimulus (continuous or non-continuous) that was the object 
of the more significant processing. The introduction of four categories of text format allows the still relatively 
small number of print-based tasks that require integration of information across formats or across texts to be 
classified respectively as mixed or multiple. 
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Tgble 1.5 Approximate distribution of tasks by text format for PISA 



Text format 


% of total tasks PISA 2009: print 


% of total tasks PISA 2009: 
electronic 


Continuous 


60 


10 


Non-continuous 


30 


10 


Mixed 


5 


10 


Multiple 


5 


70 


TOTAL 


100 


100 



In comparison, in the electronic-medium assessment, the proportion of tasks based on multiple texts is much 
greater. Given the focus of the electronic reading assessment on hypertext, in almost all units the stimulus will 
consist of multiple texts, and the tasks will require users to read across several texts (which may be different 
websites or different pages belonging to the same website), each presented in a variety of formats including 
prose paragraphs, menu lists, diagrams and other graphics. A large majority of tasks is therefore categorised 
as multiple texts for the text format variable. The relatively small numbers of tasks in the electronic reading 
assessment that require only local processing of single texts - whether they are continuous, non-continuous or 
mixed - are classified accordingly. 

Text type 

A different categorisation of text is by text type: description, narration, exposition, argumentation, instruction and 
transaction. In previous versions of the reading framework, these text types were located as subcategories of the 
continuous text format. In this new version it is acknowledged that non-continuous texts (and the elements of 
mixed and multiple texts) also have a descriptive, narrative, expository, argumentative or instructional purpose. 

Texts as they are found in the world typically resist categorisation, as they are usually not written with text 
type rules in mind, and tend to cut across categories. For example, a chapter in a textbook might include some 
definitions (exposition), some directions on how to solve particular problems (instruction), a brief historical 
account of the discovery of the solution (narration), and descriptions of some typical objects involved in the 
solution (description). The distinctions in the electronic medium are even more blurred, especially in the web 
environment where the definition of where the text begins and ends is itself contestable, and any page of 
material typically includes not only a range of text types, but also different representations that may include 
words, images, animations, video and audio files. Nevertheless, in an assessment like PISA it is useful to 
categorise texts according to the text type, based on the predominant characteristics of the text, in order to 
ensure that the instrument samples across a range of texts that represent different types of reading. 

The following classification of texts used in PISA is adapted from the work of Werlich (1 976). 

Description is the type of text where the information refers to properties of objects in space. The typical 
questions that descriptive texts provide an answer to are what questions. Descriptions can take several forms. 
Impressionistic descriptions present information from the point of view of subjective impressions of relations, 
qualities, and directions in space. Technical descriptions present information from the point of view of objective 
observation in space. Frequently, technical descriptions use non-continuous text formats such as diagrams and 
illustrations. Examples of text objects in the text type category description are a depiction of a particular place 
in a travelogue or diary, a catalogue, a geographical map, an online flight schedule or a description of a feature, 
function or process in a technical manual. 

Narration is the type of text where the information refers to properties of objects in time. Narration typically 
answers questions relating to when, or in what sequence. Why characters in stories behave as they do is 
another important question that narration typically answers. Narration can take different forms. Narratives 
present change from the point of view of subjective selection and emphasis, recording actions and events 
from the point of view of subjective impressions in time. Reports present change from the point of view of an 
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objective situational frame, recording actions and events which can be verified by others. News stories intend to 
enable the readers to form their own independent opinion of facts and events without being influenced by the 
reporter's references to his own views. Examples of text objects in the text type category narration are a novel, 
a short story, a play, a biography, a comic strip, and a newspaper report of an event. 

Exposition is the type of text in which the information is presented as composite concepts or mental constructs, 
or those elements into which concepts or mental constructs can be analysed. The text provides an explanation of 
how the different elements interrelate in a meaningful whole and often answers questions about how. Expositions 
can take various forms. Expository essays provide a simple explanation of concepts, mental constructs, or 
conceptions from a subjective point of view. Definitions explain how terms or names are interrelated with 
mental concepts. In showing these interrelations, the definition explains the meaning of words. Explications 
are a form of analytic exposition used to explain how a mental concept can be linked with words or terms. 
The concept is treated as a composite whole which can be understood by being broken down into constituent 
elements and their interrelations with each being given a name. Summaries are a form of synthetic exposition 
used to explain and communicate texts in a shorter form than the original text requires. Minutes are a record of 
the results of meetings or presentations. Text Interpretations are a form of both analytic and synthetic exposition 
used to explain the abstract concepts which are realised in a particular (fictional or non-fictional) text or group 
of texts. Examples of text objects in the text type category exposition are a scholarly essay, a diagram showing 
a model of memory, a graph of population trends, a concept map and an entry in an online encyclopaedia. 

Argumentation is the type of text that presents the relationship among concepts or propositions. Argument texts 
often answer why questions. An important subclassification of argument texts is persuasive and opinionative 
texts, referring to opinions and points of view. Comment relates the concepts of events, objects, and ideas to a 
private system of thoughts, values, and beliefs. Scientific argumentation relates concepts of events, objects, and 
ideas to systems of thought and knowledge so that the resulting propositions can be verified as valid or non-valid. 
Examples of text objects in the text type category argumentation are a letter to the editor, a poster advertisement, 
the posts in an online forum and a web-based review of a book or film. 

Instruction (sometimes called injunction) is the type of text that provides directions on what to do. Instructions 
present directions for certain behaviours in order to complete a task. Rules, regulations, and statutes specify 
requirements for certain behaviours based on impersonal authority, such as practical validity or public authority. 
Examples of text objects in the text type category instruction are a recipe, a series of diagrams showing a 
procedure for giving first aid, and guidelines for operating digital software. 

Transaction represents the kind of text that aims to achieve a specific purpose outlined in the text, such as 
requesting that something is done, organising a meeting or making a social engagement with a friend. Before 
the spread of electronic communication, this kind of text was a significant component of some kinds of letters 
and, as an oral exchange, the principal purpose of many phone calls. This text type was not included in 
Werlich's (1976) categorisation, used until now for the PISA framework. 

The term transactional is used in PISA not to describe the general process of extracting meaning from texts (as 
in reader-response theory), but the type of text written for the kinds of purposes described here. Transactional 
texts are often personal in nature, rather than public, and this may help to explain why they do not appear to 
be represented in some of the corpora used to develop many text typologies. For example, this kind of text is 
not commonly found on websites, which are frequently the subject of corpus linguistics studies (for example, 
Santini, 2006). With the extreme ease of personal communication using e-mail, text messages, blogs and social 
networking websites, this kind of text has become much more significant as a reading text type in recent years. 
Transactional texts often build on common and possibly private understandings between communicators - 
though clearly, this feature is difficult to explore in a large-scale assessment. Examples of text objects in the text 
type transaction are everyday e-mail and text message exchanges between colleagues or friends that request 
and confirm arrangements. 

Narration occupies a prominent position in many national and international assessments. Some texts are 
presented as being accounts of the world as it is (or was) and therefore claim to be factual or non-fictional. 
Fictional accounts bear a more metaphorical relation to the world as it is, appearing either as accounts of how 
it might be or of how it seems to be. In other large-scale reading studies, particularly those for school students: 
the National Assessment of Educational Progress (NAEP); the lEA Reading Literacy Study (lEARLS); and the 
lEA Programme in International Reading Literacy Study (PIRLS), the major classification of texts is between 
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fictional or literary texts, and non-fictional texts {Reading for literary experience and Reading for information 
or to perform a task in NAEP; Literary experience and Acquire and use information in PIRLS). This distinction 
is increasingly blurred as authors use formats and structures typical of factual texts in creating their fictions. 
The PISA reading literacy assessment includes both factual and fictional texts, and texts that may not be clearly 
one or the other. PISA however does not attempt to measure differences in reading proficiency between one 
type and the other. In PISA, fictional texts are classified as narration. The proportion of narrative texts in the 
print medium in PISA 2009 is similar to that in PISA 2000, at about 1 5%. Narratives in the electronic medium 
tend to be non-verbal, with animation and film having filled the position. There is therefore no specification for 
narrative in the electronic reading assessment. 



Aspect 

Whereas navigation tools and features are the visible or physical features that allow readers to negotiate their 
way into, around and between texts, aspects are the mental strategies, approaches or purposes that readers use 
to negotiate their way into, around and between texts. 

Five aspects guide the development of the reading literacy assessment tasks: 

■ retrieving information 

■ forming a broad understanding 

■ developing an interpretation 

■ reflecting on and evaluating the content of a text 

■ reflecting on and evaluating the form of a text 

As it is not possible to include sufficient items in the PISA assessment to report on each of the five aspects as 
a separate subscale, for reporting on reading literacy these five aspects are organised into three broad aspect 
categories: 

■ access and retrieve 

■ integrate and interpret 

■ reflect and evaluate 

Retrieving information tasks, which focus the reader on separate pieces of information within the text, are 
assigned to the access and retrieve scale. 

Forming a broad understanding and developing an interpretation tasks focus the reader on relationships within 
a text. Tasks that focus on the whole text require readers to form a broad understanding; tasks that focus on 
relationships between parts of the text require developing an interpretation. The two are grouped together under 
integrate and interpret. 

Tasks addressing the last two aspects, reflecting on the content of a text and reflecting on the form of a text, are 
grouped together into a single reflect and evaluate aspect category. Both require the reader to draw primarily 
on knowledge outside the text and relate it to what is being read. Reflecting on content tasks are concerned 
with the notional substance of a text; reflecting on form tasks are concerned with its structure or formal features. 

Figure 1 .3 shows the relationship between the five aspects targeted in the test development and the three broad 
reporting aspects. 
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Fi g u fe 1 . 3 ■ Relationship between the reading framework and the aspect subscales 
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An elaboration of the three broad aspect categories, encompassing tasks in both print and electronic media, is 
given below. 

Access and retrieve 

Accessing and retrieving involves going to the information space provided and navigating in that space to locate 
and retrieve one or more distinct pieces of information. Access and retrieve tasks can range from locating the 
details required by an employer from a job advertisement, to finding a telephone number with several prefix 
codes, to finding a particular fact to support or disprove a claim someone has made. 

In daily life, readers often need to retrieve information. To do so, readers must scan, search for, locate and select 
relevant information from some information space (for example, a page of continuous text, a table or a list of 
information). The required information is most frequently found in a single location, though in some cases the 
information may be in two or more sentences, in several cells of a table or in different parts of a list. 

In assessment tasks that call for retrieving information, students must match information given in the question 
with either identically worded or synonymous information in the text and use this to find the new information 
called for. In these tasks, retrieving information is based on the text itself and on explicit information included in 
it. Retrieving tasks require the student to find information based on requirements or features explicitly specified in 
questions. The student has to detect or identify one or more essential elements of a question, such as characters, 
place/time and setting, and then to search for a match that may be literal or synonymous. 

Retrieving tasks can involve various degrees of ambiguity. For example, the student may be required to select 
explicit information, such as an indication of time or place in a text or table. A more difficult version of this 
same type of task might involve finding synonymous information. This sometimes involves categorisation skills, 
or it may require discriminating between two similar pieces of information. Different levels of proficiency can 
be measured by systematically varying the elements that contribute to the difficulty of the task. 

While retrieving describes the process of selecting the required information, accessing describes the process of 
getting to the place, the information space, where the required information is located. Some items may require 
retrieving information only, especially in the print medium where the information is immediately visible and 
where the reader only has to select what is appropriate in a clearly specified information space. On the other 
hand, some items in the electronic medium require little more than accessing: for example, clicking on an 
embedded link to open a web page (in a very limited information space), or clicking to select an item in a list 
of search results. However, both processes are involved in most access and retrieve tasks in PISA. In the print 
medium such items might require readers to use navigation features such as headings or captions to find their 
way to the appropriate section of the text before locating the relevant information. In the electronic medium 
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an access and retrieve item might involve navigating across several pages of a website, or using menus, lists or 
tabs to locate relevant information. 



In both the print and electronic media, the process of accessing and retrieving information involves skills 
associated with selecting, collecting and retrieving information. Access and retrieve items in the electronic 
medium may additionally require students to navigate their way to a particular web page (for example) to find 
specified information, possibly using several navigation tools and traversing a number of pages. Students may 
be asked to click on a particular link or choose an item from a drop down menu. Examples include opening 
a website using a hyperlink; opening one or more new pages within a website; or scrolling down a page and 
clicking on a hyperlink. In accessing a particular item, students will need to make decisions about thematic 
interest. They will need to identify whether a link or site provides the information required, in terms of topical 
interest or relevance. Difficulty will be determined by several factors including the number of pages or links 
that need to be used, the amount of information to be processed on any given page, and the specificity and 
explicitness of the task directions. 



Integrate and interpret 

Integrating and interpreting involves processing what is read to make internal sense of a text. 

Interpreting refers to the process of making meaning from something that is not stated. It may involve recognising 
a relationship that is not explicit or it may be required at a more local level to infer (to deduce from evidence and 
reasoning) the connotation of a phrase or a sentence. When interpreting, a reader is identifying the underlying 
assumptions or implications of part or all of the text. A wide variety of cognitive activities is included in this 
approach. For example, a task may involve inferring the connection between one part of the text and another, 
processing the text to form a summary of the main ideas, requiring an inference about the distinction between 
principal and subordinate elements, or finding a specific instance in the text of something earlier described in 
general terms. 

/ntegrat/ngfocuses on demonstrating an understanding ofthe coherence of the text. It can range from recognising 
local coherence between a couple of adjacent sentences, to understanding the relationship between several 
paragraphs, to recognising connections across multiple texts. In each case, integrating involves connecting 
various pieces of information to make meaning, whether it be identifying similarities and differences, making 
comparisons of degree, or understanding cause and effect relationships. 

Both interpreting and integrating are required to form a broad understanding. A reader must consider the text as 
a whole or in a broad perspective. Students may demonstrate initial understanding by identifying the main topic 
or message or by identifying the general purpose or use ofthe text. Examples include tasks that require the reader 
to select or create a title or assumption for the text, explain the order of simple instructions, or identify the main 
dimensions of a graph or a table. Others include tasks that require the student to describe the main character or 
setting of a story, to identify a theme of a literary text, or explain the purpose or use of a map or figure. 

Within this aspect some tasks might require the student to identify a specific piece of text, when a theme or 
main idea is explicitly stated. Other tasks may require the student to focus on more than one part of the text - 
for instance, if the reader has to deduce the theme from the repetition of a particular category of information. 
Selecting the main idea implies establishing a hierarchy among ideas and choosing the one that is most general 
and overarching. Such a task indicates whether the student can distinguish between key ideas and minor 
details, or can recognise the main theme in a sentence or title. 

Both interpreting and integrating are also involved in developing an Interpretation, which requires readers 
to extend their initial broad impressions so that they develop a deeper, more specific or more complete 
understanding of what they have read. Many tasks in this category call for logical understanding: readers must 
process the organisation of information in the text. To do so, readers must demonstrate their understanding of 
cohesion even if they cannot explicitly state what cohesion is. In some instances, developing an interpretation 
may require the reader to process a sequence of just two sentences relying on local cohesion. This might 
even be facilitated by the presence of cohesive markers, such as the use of "first" and "second" to indicate a 
sequence. In more difficult instances (for example, to indicate relations of cause and effect), there might not be 
any explicit markings. 
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Other tasks include comparing and contrasting information, and identifying and listing supporting evidence. 
Compare and contrast tasks require the student to draw together two or more pieces of information from the 
text. In order to process either explicit or implicit information from one or more sources in such tasks, the reader 
must often infer an intended relationship or category. 

As well as these integrative tasks, developing an Interpretation tasks may involve drawing an inference from a 
more localised context: for example, interpreting the meaning of a word or phrase that gives a particular nuance 
to the text. This process of comprehension is also assessed in tasks that require the student to make inferences 
about the author's intention, and to identify the evidence used to infer that intention. 

In traditional print environments, information might be located in a single paragraph, across different paragraphs 
or sections of text, or across two or more texts. In electronic environments, integration can be more complex. 
In a web environment, for example, information can be connected in a non-sequential way through a series of 
hyperlinks. While integration may take on increased complexity in electronic environments, those environments 
also provide tools that can facilitate the integration process. For example, views in a word processing program 
can be manipulated so that information in various locations can be viewed simultaneously, facilitating 
comparisons. Individuals take on an increased responsibility to know and understand which information can 
be used and which tools can be used to view it in order to facilitate the integration of information. 

The way we synthesise information is also transformed in the electronic environment. We synthesise information 
in the print medium, of course, but typically this takes place within continuous text that has been constructed 
for us. Electronic reading is different in that readers actually construct the texts that they read by the choices 
they make in the links that they follow, collecting a series of texts and synthesising the essential aspects of each 
during the comprehension process. Synthesis is also different in that readers often skip more information than 
they read on any single page; the units of text that readers find useful on any single page are often quite small 
and they rarely read all of the information on a single web page. 

An integrate and Interpret task in the electronic medium may involve surfing several pages of a website or 
combining information from different sites, or it may require drawing inferences from information on a single 
page. As in print reading, tasks will include comparing, contrasting, finding evidence, determining influence, 
generalising and analysing subtleties of language. Difficulty will be determined by several factors including the 
number of pieces of information to be integrated and the number of locations where they are found, as well as 
the verbal complexity and the subject familiarity of the textual information. 

As mentioned above. Interpreting signifies the process of making meaning from something that is not explicitly stated. 
In recognising or identifying a relationship that is not explicit, an act of interpretation is required: thus interpretation 
is perhaps always involved somewhere in the process of Integration as described above. The relationship between 
the processes of integration and interpretation may therefore be seen as intimate and interactive. Integrating involves 
first inferring a relationship within the text (a kind of interpretation), and then bringing pieces of information together, 
therefore allowing an interpretation to be made that forms a new integrated whole. 

Reflect and evaluate 

Reflecting and evaluating involves drawing upon knowledge, ideas or attitudes beyond the text in order to relate 
the information provided within the text to one's own conceptual and experiential frames of reference. 

Reflect items may be thought of as those that require readers to consult their own experience or knowledge to 
compare, contrast or hypothesise. Evaluate items are those that ask readers to make a judgment drawing on 
standards beyond the text. 

Reflecting on and evaluating the content of a text requires the reader to connect information in a text to knowledge 
from outside sources. Readers must also assess the claims made in the text against their own knowledge of the world. 
Often readers are asked to articulate and defend their own points of view. To do so, readers must be able to develop 
an understanding of what is said and intended in a text. They must then test that mental representation against what 
they know and believe on the basis of either prior information, or information found in other texts. Readers must call 
on supporting evidence from within the text and contrast it with other sources of information, using both general and 
specific knowledge as well as the ability to reason abstractly. 
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Assessment tasks representing this category of processing include providing evidence or arguments from outside 
the text, assessing the relevance of particular pieces of information or evidence, or drawing comparisons with 
moral or aesthetic rules (standards). The task might require a student to offer or identify alternative pieces of 
information to strengthen an author's argument, or evaluate the sufficiency of the evidence or information 
provided in the text. 



The outside knowledge to which textual information is to be connected may come from the student's own 
knowledge or from ideas explicitly provided in the question. In the PISA context, any outside knowledge 
required is intended to be within the expected range of 15-year-olds' experiences. For example, it is assumed 
that 1 5-year-olds are likely to be familiar with the experience of going to the movies, a context that is drawn 
upon in the items related to the stimulus Macondo, discussed below. 



Reflecting on and evaluating the form of a text requires readers to stand apart from the text, to consider it 
objectively and to evaluate its quality and appropriateness. Implicit knowledge of text structure, the style typical 
of different kinds of texts and register play an important role in these tasks. These features, which form the basis 
of an author's craft, figure strongly in understanding standards inherent in tasks of this nature. Evaluating how 
successful an author is in portraying some characteristic or persuading a reader depends not only on substantive 
knowledge but also on the ability to detect subtleties in language -for example, understanding when the choice 
of an adjective might influence interpretation. 



Some examples of assessment tasks characteristic of reflecting on and evaluating the form of a text include 
determining the usefulness of a particular text for a specified purpose and evaluating an author's use of 
particular textual features in accomplishing a particular goal. The student may also be called upon to describe 
or comment on the author's use of style and to identify the author's purpose and attitude. 



While the kinds of reflection and evaluation called for in the print medium assessment are also required in the 
electronic medium, evaluation in the electronic medium takes on a slightly different emphasis. 



Printed texts are typically edited and filtered by many layers of the print publication process. On the web, 
however, anyone can publish anything. Moreover, the homogeneity of electronic text formats (windows, 
frames, menus, hyperlinks) tends to blur the distinctions across text types. These new features of electronic text 
increase the need for the reader to be aware of authorship, accuracy, quality and credibility of information. 
Whereas in the printed text clear indications of the source are usually available (for example, through the 
standard practice of mentioning author - sometimes with credentials, publisher, date and place of publication 
in the book), in electronic texts that important information is not always available. There is therefore a 
need for readers of electronic text to be more active in assessing and reasoning about source features. As 
people have access to a broadening universe of information in networked environments, evaluation takes 
on an increasingly critical role. While published print information carries a sense of legitimacy stemming 
from the assumed reviewing and editing process, sources for online information are more varied, ranging 
from authoritative sources to postings with unknown or uncertain authenticity. Such information must be 
evaluated in terms of accuracy, reliability and timeliness. It is important to recognise that the evaluation 
process happens continuously and therefore is a critical component of electronic reading. Once a reader 
has located information related to the question or problem, critical analysis of that information becomes 
important. While critical analysis of information takes place in the print medium, of course, it is even more 
important on line. Moreover, it is a skill that few adolescents appear to possess; they are easily fooled by 
false information appearing on the web and do not always possess strategies to analyse its accuracy (see for 
example Leu & Castek, 2006). 



Skilled readers in the electronic medium know how to evaluate information that may be questionable. They also 
know how to use a search engine to gather additional information about a site by simply conducting a search 
for its title and additional words such as "hoax," "true?" or "accurate?" Critical analysis, a skill required during 
print reading comprehension, is transformed in important ways in the electronic medium, requiring new online 
reading skills. For instance, the availability of Internet resources in schools has encouraged teachers to include 
document search assignments in their instructional strategies. In order to complete those assignments, students 
need to be not only good at understanding what they read, but also at searching. This can be compared with a 
more traditional document-based teaching session where a single document would be pre-selected, copied and 
distributed by the teacher for in-depth reading and study. 
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Rieh (2002) identifies two distinct kinds of judgement that Internet users tend to make: predictive judgements 
and evaluative judgements. A reflect and evaluate item may involve making a predictive judgement about which 
site to go to from a range of possibilities, based on relevance, authenticity and authority. Once at a site, a reader 
completing a reflect and evaluate item may need to make an evaluative judgement: students may be asked 
to evaluate a site or link in terms of authority or reliability, credibility and trustworthiness of information. For 
example, they may need to assess whether the information is official and authoritative, unsupported personal 
opinion, or propaganda designed to influence the reader. 

Some items may require reflection on and evaluation of the content of a site in similar ways to those currently 
used in print reading: for example, where students give a personal response to ideas and opinions, using 
background knowledge and experience. As with print assessment items, the difficulty of reflect and evaluate 
items is determined by several factors including the quantity and explicitness of information to support a 
reflection and evaluation and the extent to which the information is common knowledge. 

To some extent every critical judgment requires the reader to consult his or her own experience; some kinds 
of reflection, on the other hand, do not require evaluation (for example, comparing personal experience with 
something described in a text). Thus evaluation might be seen as a subset of reflection. 

The aspects of reading in print and electronic media 

The three broad aspects defined for PISA reading literacy are not conceived of as entirely separate and 
independent, but rather as interrelated and interdependent. Indeed from a cognitive processing perspective 
they can be considered semi-hierarchical: it is not possible to interpret or integrate information without having 
first retrieved it, and it is not possible to reflect on or evaluate information without having made some sort 
of interpretation. In PISA, however, the focus is on developing an assessment framework that will guide the 
construction of an assessment to operationalise and subsequently measure proficiency in different aspects of 
the reading domain. The framework description of reading aspects distinguishes approaches to reading that are 
demanded for different contexts and purposes; these are then reflected in assessment tasks that emphasise one 
or other aspect. All readers, irrespective of their overall proficiency, are expected to be able to demonstrate 
some level of competency in each of the reading aspects (Langer, 1995), since all are seen as being in the 
repertoire of each reader at every developmental level. 

Given that the aspects are rarely if ever entirely separable, the assignment of a task to an aspect is often a matter 
of fine discrimination that involves judgements about the salient (most important) features of the task, and about 
the predicted typical approach to it. Figure 1 .4 and Figure 1 .5 represent the way the aspects are operationalised 
in different tasks, in print and electronic media respectively. The boxes around aspect names represent the 
emphasis of the task, while the presence of the other aspects at each task point acknowledge that all the aspects 
(as cognitive processes) are likely to play some role in each task. 

For the most part, identifying the aspect for each PISA reading literacy task - the task being the question or 
directive that the student sees - will depend on the objective of the task. For example, retrieving a single piece 
of explicitly stated information from a web page (such as finding out the number of Internet users worldwide) 
may involve a complex series of steps requiring the evaluation of the relevance of several results on a search 
result page, comparing and contrasting descriptions and deciding which of several sources is likely to be 
most authoritative. Nevertheless, if the ultimate task, finding the number of Internet users worldwide, is stated 
explicitly once the target page has been reached, this task is classified as access and retrieve. This is the 
approach that has been taken in PISA print reading to classify each task by aspect. 



PISA 2009 ASSESSMENT FRAMEWORK - KEY COMPETENCIES IN READING, MATHEMATICS AND SCIENCE 



OECD 2009 



PISA 2009 READING FRAMEWORK 



Fi g u f e 1 . 4- ■ Relationship between task, text and aspects in the print medium 
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Complex electronic reading tasks - simulating the complexity of real-life reading 

In both the print and electronic media, real-life reading tasks can typically involve searching for information in 
what is essentially an unlimited space. In the print world, we might go to a library, search a catalogue, browse 
on the library shelf, and then, having found what looks like the right book or books, scan the contents and 
flip through pages before making a selection of one or many resources. In the electronic medium, the parallel 
process is experienced sitting at a computer screen and accessing a database or the Internet. This is a much 
more contained task in geographical space and time, but the processes of sorting, selecting, evaluating and 
integrating are in many respects similar to searching for information in the print medium. Practical constraints 
have meant that reading assessments like PISA cannot measure students' proficiency in searching for print 
resources. Consequently, the large-scale assessment of reading has generally confined itself, until now, to tasks 
that involve reading rather short and clearly designated texts. By contrast, large scale assessments can, in the 
electronic medium, authentically measure proficiency in accessing, selecting and retrieving information from 
a wide range of possible resources. Therefore, insofar as the cognitive processing in the two media is parallel, 
an assessment of electronic reading makes it possible to measure something that has until now, not been 
measurable in a large-scale assessment. 
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Fig u re 1 .5 ■ Relationship between task, text and aspects in the electronic medium 
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One important point made about the difference between print and electronic reading is the lack of obvious 
direction provided to the reader in the electronic medium. In the print medium the arrangement and organisation 
of (a single) text is given to the reader by the author's ordering of pages, chapters or larger sections. The reader 
may choose to take a different path, but a path is nevertheless suggested by the physical arrangement of the 
text. In hypertext on the other hand the reader has no direction - or, to put it more positively, the reader has far 
greater control of the selection and sequence of reading. In effect, control is forced upon the reader. 

In order to simulate the real-life freedom (or forced independence) of reading in the electronic medium, some 
part of electronic reading assessment is designed in the form of complex reading tasks. Here not only is the 
range of texts comparatively undefined, but also the optimal sequence of steps to be taken in completing the 
task is not pre-determined. These are tasks which take advantage of the compression and economy of electronic 
reading resources. They also require the use of the variety of skills: accessing and retrieving; interpreting and 
integrating; and reflecting and evaluating. The task cannot therefore be assigned to a single aspect. The equal 
status of the three aspects, and their relationship to dynamic reading texts, is represented in Figure 1 .6. 
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Fig u re 1 .6 ■ Relationship between task, text and aspects in complex reading tasks 
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While the three aspects do not usually operate entirely independently of one another in either print or electronic 
reading tasks, it is possible to construct relatively simple tasks in which there is a clear emphasis on one or the 
other aspect. In complex tasks, on the other hand, the process is not so well defined. The reader assimilates the 
task, and then confronts the problem of interpreting, extrapolating from and evaluating the immediately visible 
text (for example, the home page of a website) to find relevant information. In order to do this an experienced 
reader in the electronic medium uses knowledge about access structures and typical site structures (just as 
an experienced print reader uses knowledge about generic textual structures and features). It is not just a 
matter of knowledge however: a skilled reader of electronic texts also needs to theorise and construct logical 
and interpretive connections between seen and unseen multiple texts. In an authentic, complex task in the 
electronic medium, the reader needs to process the visible information immediately and extrapolate from it: 
making judgments, synthesising and accessing information in an integrated, recursive sequence. 

In order to maintain trend and represent the balance of the aspects applicable in the assessment environment, 
the distribution by aspect across the print-based assessment remains similar to that of PISA 2000. In the 
electronic reading assessment, however, the tasks are spread more equally across the three traditional aspects 
and the new composite aspect, complex. The approximate distribution of tasks by aspect in print and electronic 
media is shown in Table 1 .4. 
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Tgble 1.4 Approximate distribution of tasks by aspect for PISA 2009 



Aspect 


% of total tasks PISA 2009: print 


% of total tasks PISA 2009: electronic 


Access and retrieve 


25 


25 


Integrate and interpret 


50 


35 


Reflect and evaluate 


25 


20 


Complex 


0 


20 


TOTAL 


100 


100 



Summary of the relationship between printed and electronic reading texts and tasks 

Table 1 .5 presents some of the esessential similarities and differences between print and electronic reading. One 
purpose of the table is to describe intrinsic similarities and differences between print reading and electronic 
reading. In many cases the entries under Print reading and Electronic reading are identical. In other places, 
the descriptions highlight some essential differences between reading in the two media. 

A second purpose of the table is to illustrate similarities and differences in what PISA assesses in the two 
media. In some cases it is a matter of prominence and emphasis: square brackets signify that a feature is given 
relatively little emphasis in the PISA assessment. In other cases the difference is more absolute. While some 
features exist in both media, they cannot be or are not assessed in PISA. These are represented in grey scale. 
One of the principles in constructing the PISA frameworks and the assessment tasks that operationalise them 
is to represent the domains authentically. There is no set way of doing this, and in a sense the decisions and 
selections made are arbitrary - though based on the best judgment of international reading experts. How 
the domain is described and operationalised, in this and other respects, is determined by a combination of 
conceptual, empirical and political considerations. The aim in the scoping of the domain outlined above is to 
explain the basis for building an assessment for PISA 2009 that captures the essence of reading literacy. Such 
an assessment will in turn yield an array of data from which to report 1 5-year-olds' reading proficiency in ways 
that are comprehensive, meaningful and relevant. 
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1 Tgblel 


5 Similarities and differences between print and electronic reading, by main 


1 


framework characteristics 




Print reading 


Electronic reading 


Situations 


Personal 


Personal 




Public 


Public 




Occupational 


Occupational 




Educational 


Educational 


Texts: Environments not applicable 


Authored 






Message based 


Texts: Formats 


Continuous 


[Continuous] 




Non-continuous 


[Non-continuous] 




[Mixed] 


[Mixed] 




[Multiple] 


Multiple 


Texts: Text type 


Argumentation 


Argumentation 




Description 


Description 




Exposition 


Exposition 




Narration 


Narration 




Instruction 


Instruction 




Transaction 


Transaction 


Aspects (1 ) 


Access and retrieve 


Access and retrieve 




Search 


Search 




Orient and navigate in concrete 


Orient and navigate in abstract 




information space 


information space 




e.g. Co to library, search in a catalogue, find 
a book 


e.g. Enter URL; user search engines 




Use navigation tools and structures 


Use navigation tools and structures 




e.g. Table of contents; page numbers: 
glossary 


e.g. Menus; embedded hyperlinks 




Select and sequence information 


Select and sequence information 




- low reader control 


- high reader control 




- one sequence of linear reading 


- multiple sequences of linear reading 


Aspects (2) 


Integrate and interpret 


Integrate and interpret 




Integrate at a lower level of demand: 


Integrate at a higher level of demand: 




larger portions of text are simultaneously 


limited parts of text are simultaneously 




visible 


visible 




(one or two pages) 


(limited by screen size) 




Develop an interpretation 


Develop an interpretation 




Form a broad understanding 


Form a broad understanding 


Aspects (3) 


Reflect and evaluate 


Reflect and evaluate 




Pre-evaluate information 


Pre-evaluate information 




e.g. use table of contents; skim passages. 


e.g. use menus; skim web pages, checking 




checking for credibility and usefulness 


for credibility and usefulness 




[Evaluate credibility of source 


Evaluate credibility of source 




- usually less important due to filtering and 


- usually more important due to lack 




preselection in the publishing process] 


of filtering and preselection in open 
environment 




Evaluate plausibility of content 


Evaluate plausibility of content 




Evaluate coherence and consistency 


Evaluate coherence and consistency 




Hypothesise 


Hypothesise 




Reflect in relation to personal experience 


Reflect in relation to personal experience 


Aspects (4) 


Complex 


Complex 




The range of sources to be consulted is 


The range of sources to be consulted is 




relatively undefined 


relatively undefined 




The sequence of steps within the task 


The sequence of steps within the task 




is undirected 


is undirected 




e.g. finding, evaluating and integrating 


e.g. finding, evaluating and integrating 




information from multiple printed texts 


information from multiple electronic texts 
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ASSESSING READING LITERACY 

The previous section outlined the conceptual framework for reading literacy. The concepts in the framework 
must in turn be represented in tasks and questions in order to collect evidence of students' proficiency in 
reading literacy. 

Building tasks in the print medium 

The distribution of tasks across the major framework variables of situation, text and aspect was discussed in 
the previous section. In this section some of the other major issues in constructing and operationalising the 
assessment are considered: factors affecting item difficulty, and how difficulty can be manipulated; the choice 
of response formats; and some issues around coding and scoring. 

Factors affecting item difficulty 

The purpose of the PISA reading literacy assessment is to monitor and report on the reading proficiency of 
15-year-olds as they approach the end of compulsory education. Each task in the assessment is designed to 
gather a specific piece of evidence about that proficiency by simulating a reading activity that a reader might 
carry out either inside or outside school, as an adolescent or as an adult. 

The PISA reading literacy tasks range from very straightforward comprehension activities to quite sophisticated 
activities requiring deep and multiple levels of understanding. The difficulty of any reading literacy task depends 
on an interaction amongst several variables. Drawing on Kirsch and Mosenthal's work (see for example Kirsch, 
2001; Kirsch & Mosenthal, 1990), we can manipulate the difficulty of items by applying knowledge of the 
following aspect and text format variables. 

In access and retr/eve tasks, difficulty is conditioned by the number of pieces of information that the reader needs 
to locate, by the amount of inferencing required, by the amount and prominence of competing information, and 
by the length and complexity of the text. 

In integrate and interpret tasks, difficulty is affected by the type of interpretation required (for example, making 
a comparison is easier than finding a contrast); by the number of pieces of information to be considered; by the 
degree and prominence of competing information in the text; and by the nature of the text: the less familiar and 
the more abstract the content and the longer and more complex the text, the more difficult the task is likely to be. 

In reflect and evaluate tasks, difficulty is affected by the type of reflection or evaluation required (from least to 
most difficult, the types of reflection are: connecting; explaining and comparing; hypothesising and evaluating); 
by the nature of the knowledge that the reader needs to bring to the text (a task is more difficult if the reader 
needs to draw on narrow, specialised knowledge rather than broad and common knowledge); by the relative 
abstraction and length of the text; and by the depth of understanding of the text required to complete the task. 

In tasks relating to continuous texts, difficulty is influenced by the length of the text; the explicitness and 
transparency of its structure; how clearly the parts are related to the general theme; and whether there are text 
features such as paragraphs or headings, and discourse markers such as sequencing words. 

In tasks relating to non-continuous texts, difficulty is influenced by the amount of information in the text; the 
list structure (simple lists are easier to negotiate than more complex lists); whether the components are ordered 
and explicitly organised, for example with labels or special formatting; and whether the information required is 
in the body of the text or in a separate part, such as a footnote. 

Response formats 

The form in which the evidence is collected - the response format - varies according to what is considered 
appropriate given the kind of evidence that is being collected, and also according to the pragmatic constraints 
of a large-scale assessment. As in any large-scale assessments the range of feasible item formats is limited, with 
multiple-choice and short constructed-response items (where students write their own answer) being the most 
manageable formats. 
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Several studies based on PISA data suggest that response format has a significant effect on the performance of 
different groups: for example, students at different levels of proficiency (Routitsky & Turner, 2003); students in 
different countries (Grisay & Monseur, 2007); and boys and girls (Lafontaine & Monseur, 2006). Given this 
variation, Lafontaine and Monseur caution that in measuring trends over time, it is important to maintain a 
similar proportion of tasks in multiple choice and constructed response formats from one administration to 
the next. A further significant consideration in the context of reading literacy is that open constructed response 
items are particularly important for the reflection and evaluation aspect, where the intent is often to assess 
the quality of thinking rather than the conclusion itself. Finally, students in different countries are more or less 
familiar with various response formats. Including items in a variety of formats is likely to provide some balance 
between more and less familiar formats for all students, regardless of nationality. 



In summary, to ensure proper coverage of the ability ranges in different countries, to ensure fairness given the 
inter-country and gender differences observed, and to ensure a valid assessment of the reflect and evaluate 
aspect, both multiple choice and open constructed response items continue to be used in PISA reading 
literacy assessments. Any major change in the distribution of item types in print reading might also impact on 
the measurement of trends. In the interests of economy, however, and to take advantage of the capacity for 
automated coding that the electronic medium offers, for the electronic reading assessment a higher proportion 
of items requiring no coder judgement has been included. 



Table 1 .6 and Table 1 .7 show coding requirements for print and electronic tasks respectively. The distribution 
is shown in relation to the three aspects of reading literacy for print literacy and the four aspects described for 
the electronic reading assessment. Items that require expert judgement consist of open constructed and short 
constructed responses. Items that do not require coder judgement consist of multiple choice, complex multiple 
choice and closed constructed response items. The closed constructed response items are those that require 
the student to generate a response, but require minimal judgment on the part of a coder. For example, a task 
in which a student is asked to copy a single word from the text, where only one word is acceptable, would be 
classified as a closed constructed response item. Such items impose a minor cost burden in operational terms 
and therefore from a pragmatic perspective, these closed constructed response items can be grouped with 
multiple choice items. 



Tgble 16 Approximate distribution of tasks by coding requirement for PISA 2009: print medium 



Aspect 


% of tasks requiring expert 
judgment m coding 


% of tasks not requiring expert 
judgment in coding 


% of test 


Access and retrieve 


11 


14 


25 


Integrate and interpret 


14 


36 


50 


Reflect and evaluate 


18 


7 


25 


TOTAL 


43 


57 


1 00 



46 
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Tgble 1 7 Approximate distribution of tasks by coding requirement for PISA 2009: 

electronic medium 



Aspect 


% of tasks requiring expert 
judgment in coding 


% of tasks not requiring expert 
judgment in coding 


% of test 


Access and retrieve 


0 


25 


25 


Integrate and interpret 


0 


35 


35 


Reflect and evaluate 


15 


5 


20 


Complex 


15 


5 


20 


TOTAL 


30 


70 


100 



The tables indicate that while there is some distribution of items that require coder judgement and those that do 
not across the aspects, they are not distributed evenly. The reflection and evaluation aspect in both media and 
the complex tasks in the electronic reading assessment are assessed through a larger percentage of constructed 
response items, which require expert coder judgment. 

Coding and scoring 

Codes are applied to test items, either by a more or less automated process of capturing the alternative chosen 
by the student for a multiple choice answer, or by a human judge (expert coder) selecting a code that best 
captures the kind of response given by a student to an item that requires a constructed response. The code, of 
either type, is then converted to a score for the item. Scoring is relatively simple with multiple-choice items or 
other closed response format items such as selecting an item from a drop down menu in the electronic medium: 
the student has either chosen the designated correct answer or not, so the item is scored as 1 or 0 respectively. 

Partial-credit models allow for more complex scoring of constructed response items. Some answers, even though 
incomplete, are better than others. Given that incomplete answers indicate a higher level of reading literacy 
than inaccurate or incorrect answers, they receive partial credit. Such items are then scored polytomously - that 
is, there is a full credit score, one or more partial credit scores, and a no credit score. Psychometric models for 
such polytomous scoring are well established and in some ways are preferable to dichotomous scoring (full 
credit or no credit), as they use more of the information in the responses. Interpretation of polytomous scoring 
is more complex, however, as each task will have several locations on the difficulty scale: one for the full-credit 
answer and others for each of the partial-credit answers. Partial-credit scoring is used for some of the more 
complex constructed response items in PISA. 

There is a great range of constructed response tasks. Some require little judgement on the coder's part; these 
include tasks that ask the reader to simply mark parts of the text to indicate an answer or to list a few words. 
Others require considerable judgement on the part of coder, as for example when the reader is asked to explain 
the main idea of a text in his or her own words. 

The following three examples illustrate some of the salient variables addressed in the construction of print 
reading items, including situation, text format, text type, and aspect. They also illustrate a range of response 
formats and coding and scoring features. The three items were part of the PISA reading literacy instrument 
administered in the field trial for PISA 2009. The first and third were also administered in the PISA main survey 
in the year 2000. 
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PRINT READING UNIT 1 

MACONDO 

Dazzled by so many and such marvellous inventions, the people of Macondo did not know where their 
amazement began. They stayed up all night looking at the pale electric bulbs fed by the plant that Aureliano 
Triste had brought back when the train made its second trip, and it took time and effort for them to grow 
accustomed to its obsessive toom-toom. They became indignant over the living images that the prosperous 
merchant Don Bruno Crespi projected in the theatre with the lion-head ticket windows, for a character who 
had died and was buried in one film, and for whose misfortune tears of affliction had been shed, would 
reappear alive and transformed into an Arab in the next one. The audience, who paid two centavos apiece 
to share the difficulties of the actors, would not tolerate that outlandish fraud and they broke up the seats. 
The mayor, at the urging of Don Bruno Crespi, explained by means of a proclamation that the cinema was 
a machine of illusions that did not merit the emotional outburst of the audience. With that discouraging 
explanation many felt that they had been the victims of some new and showy gypsy business and they 
decided not to return to the movies, considering that they already had too many troubles of their own to 
weep over the acted-out misfortunes of imaginary beings. 

Macondo is a piece of prose from the novel One Hundred Years of Solitude by the Colombian author Gabriel 
Garcia Marquez. It is classified as belonging to the personal situation because it was written for readers' interest 
and pleasure. The Macondo unit in PISA is introduced with a brief paragraph to orientate the reader: "The 
passage on the opposite page is from a novel. In this part of the story, the railway and electricity have just been 
introduced to the fictional town of Macondo, and the first cinema has opened." The people's reaction to the 
cinema is the focus of the passage. While the historical and geographical setting of the extract is exotic for most 
readers, going to the movies is within the experience of 15-year-olds, and the characters' responses are at the 
same time intriguing and humanly familiar. Within the continuous text format category, Macondo is an example 
of narrative writing in that it shows, in a manner typical of this text type, why characters in stories behave as they 
do, recording actions and events from the point of view of subjective impressions. 

The Macondo unit comprises tasks covering the aspects of integrate and interpret and reflect and evaluate. One 
of the tasks is reproduced below. Further tasks can be found in Annex A1 . The numbering of the tasks is identical 
to the numbering used in the test booklets given to students. 

Question 3: MACONDO 

At the en4 of the p^ssgqe, why 4i4 the people of M3con4o 4ed4e not to return to the moviesi’ 

A. They wgnte4 amusement gn4 4istrgction, but foun4 thgt the movies were realistic 
gn4 4epressing. 

B. They coul4 not gffor4 the ticket prices. 

C. They wgnte4 to sgve their emotions for regl-life occasions. 

D. They were seeking emotional involvement, but foun4 the movies boring, 
unconvincing gn4 of poor quality. 



The correct answer is C. 

This task requires integrating and interpreting to form a broad understanding. In order to gain credit, students 
need to synthesise elements across the text to identify the reason that the characters in the story behaved as they 
did at the end. In selecting option C, they must reject some reasons that could plausibly explain why people 
might decide not to go to the movies, represented by distractors that are based on preconceptions rather than 
on the text. 
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Table 1 .8 summarises the framework characteristics of the Macondo question. 



T^ble 1.8 Framework characteristics of sample task: Macondo 



Situation 


Personal 


Text format 


Continuous 


Text type 


Narration 


Aspect 


Integrate and interpret: Form a broad understanding 


Item format 


Multiple choice 
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PRINT READING UNIT 2 

METROTRANSIT 




50 
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The Metrotransit unit from the PISA 2009 field trial presents a graphic from a public document, a map of an 
urban transport network in the form of a diagram. It uses fictional place names that participating countries were 
invited to adapt for their national versions. The text is non-continuous; it could be presented as a combined 
list of stations categorised according to line, features and grid locations. Though relatively simple, it includes 
a complicating element - a key of symbols - the application of which is required to gain full credit for the 
question reproduced below. 



Question 4: METROTRANSIT 

You rtee4 to ffn4 the shortest route by urt4erqrourt4 t 3 il from Sgto Station to Forest Station. 

Prgw on the mgp the route thgt you woul4 tgke. 

This access and retrieve task requires students to locate and sequence multiple pieces of information - the 
names of stations - to plan a route. The task simulates a real-life experience that demands careful reading 
of a non-continuous text. The item was constructed to ensure that use of conditional information - that is, 
information external to the main part of a text - must be processed in order to complete the task successfully. 
For full credit, students therefore need to use the information provided in the key: that part of the system that is 
under construction. Students who take account of this conditional information realise that what is apparently 
the shortest route is currently unusable. The requirement to use conditional information increases the difficulty 
of items significantly. Only about one-third of students in the field trial gained full credit for this item. However 
more than half of the other students, who plotted a route using stations on Line 5, were given partial credit, as 
they demonstrated that in all but one respect (albeit a crucial one, in practical terms) they had understood and 
could use information that they had retrieved. Examples of full and partial credit responses are shown below. 

Full credit response 




PISA 2009 ASSESSMENT FRAMEWORK - KEY COMPETENCIES IN READING, MATHEMATICS AND SCIENCE © OECD 2009 



PISA 2009 READING FRAMEWORK 



Partial credit response 




Most of the items in PISA print reading require a multiple choice or text response. The short response format 
of this question, involving the drawing of a route on a map, illustrates that the range of response types can 
nevertheless vary to some extent, according to the format that seems most appropriate for the particular task. 
This is bounded, of course, by the practical limitations of a large-scale paper and pen test. 

Table 1 .9 summarises the framework characteristics of the Metrotransit question. 

T^ble 19 Framework characteristics of sample task: Metrotransit 






Situation 


Public 


Text format 


Non-continuous 


Text type 


Description 


Aspect 


Access and retrieve: Retrieve information 


Item format 


Short response 
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PRINT READING UNIT 3 

STUDENT OPINIONS 



There are so many people out there dying from hunger and disease, yet 
we are more concerned about future advancements. We are leaving these 
people behind as we attempt to forget and move on. Billions of dollars are 
poured into space research by large companies each year. If the money 
spent on space exploration was used to benefit the needy and not the 
greedy, the suffering of millions of people could be alleviated. 



Ana 

The challenge of exploring space is a source of inspiration for many 
people. For thousands of years we have been dreaming of the heavens, 
longing to reach out and touch the stars, longing to communicate with 
something we only imagine could exist, longing to know... Are we alone? 

Space exploration is a metaphor for learning, and learning is what drives 
our world. While realists continue to remind us of our current problems, 
dreamers stretch our minds. It is the dreamers' visions, hopes and desires 
that will lead us into the future. 



Beatrice 

We ruin rain forests because there is oil under them, put mines in sacred 
ground for the sake of uranium. Would we also ruin another planet for the 
sake of an answer to problems of our own making? Of course! 

Space exploration strengthens the dangerous belief that human problems 
can be solved by our ever-increasing domination of the environment. 
Human beings will continue to feel at liberty to abuse natural resources 
like rivers and rain forests if we know there is always another planet 
around the corner waiting to be exploited. 

We have done enough damage on Earth. We should leave outer space alone. 

Dieter 

The earth's resources are quickly dying out. The earth's population is 
increasing at a dramatic rate. Life cannot be sustained if we continue to 
live in such a way. Pollution has caused a hole in the ozone layer. Fertile 
lands are running out and soon our food resources will diminish. Already 
there are cases of famine and disease caused by over-population. 

Space is a vast empty region which we can use to our benefit. By supporting 
exploration into space, one day we may find a planet that we can live on. 
At the moment this seems unimaginable, but the notion of space travel 
was once thought of as impossible. Discontinuing space exploration in 
favour of solving immediate problems is a very narrow-minded and short- 
term view. We must learn to think not only for this generation but for the 
generations to come. 



Felix 
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To ignore what the exploration of space has to offer would be a great 
loss to all mankind. The possibilities of gaining a greater understanding 
of the universe and its beginnings are too valuable to waste. The study 
of other celestial bodies has already increased our understanding of our 
environmental problems and the possible direction Earth could be heading 
in if we don't learn to manage our activities. 

There are also indirect benefits of research into space travel. The creation of 
laser technology and other medical treatments can be attributed to space 
research. Substances such as teflon have come out of mankind's quest to 
travel into space. Thus new technologies created for space research can 
have immediate benefits for everyone. 



Kate 

The stimulus for the unit Student Opinions consists of five short argumentative texts that offer opinions about 
space research. Because it is based on writing by students in their final year of school, the text is classified as 
educational in terms of situation. All of the short pieces that make up the stimulus for this unit are continuous, 
but they were generated independently, and are juxtaposed for the purposes of the assessment, the format 
category is multiple texts. The stimulus is classified as argumentation, as the texts set forth propositions and 
attempt to persuade the reader to a point of view. 

Several of the tasks that were based on this stimulus were Integrate and interpret questions that represent typical 
active reading, as they simulate the processes that readers might engage in as they compare and contrast the 
opinions of different authors on a topic. Another typical active reading approach in encountering one or more 
written arguments is to compare one's own position with those of the writers. The sample task that follows 
represents that kind of reflective and evaluative reading activity. Further tasks can be found in Annex At . The 
numbering of the tasks is identical to the numbering used in the test booklets given to students. 



Question 6: STUDENT OPINIONS 

Thinking gbout the mgin icjegs presentee! by the five stuejents, which stu4ent <\o you ggree with most 
stronglyf 

Student's ngme; 

Wsing your own words, explain your choice by referring to your own opinion and the main ideas 
presented by the student. 



This task requires students to draw on their own knowledge and beliefs to evaluate the arguments put forward 
by the writers, comparing the substance rather than the form of the texts. In the five-aspect categorisation, this 
task is therefore classified as reflecting on the content of a text. In order to gain credit for this item, students 
needed to demonstrate implicitly or explicitly that they understood the main thrust of the argument advanced 
by their chosen writer, as well as justifying their position, either by introducing their own supporting argument 
or by summarising or interpreting the argument given by the writer. 

Full credit was available regardless of which writer was nominated by the student, as long as the criteria 
outlined above were satisfied. Some typical answers that earned full credit were: 

o Ana I feel that we should take care of what Is going on In our own world before we blow all our money 
on space exploration. I understand the Importance of some exploration but I think disease and famine need 
to be helped out of this world first. 



54 
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Dieter I agree with him because he is concerned with the environment and he thinks that we should leave 
outer space alone. 

o Felix I agree with Felix because unless we are willing to face extinction, there is no other place to go after 
we have wrecked the earth. 




Answers that were not given credit often quoted from one of the texts without addition, whereas the task 
directed students to use their own words. Other responses that gained no credit were vague, or offered a 
general opinion without substantiating it with reference to one of the texts (for example, "Dieter. Let's look at 
the realities. ). 

Table 1 .1 0 summarises the framework characteristics of the Student Opinions question. 



Tgble 1.10 Framework characteristics of sample task: Student Opinions 



Situation 


Educational 


Text format 


Multiple 


Text type 


Argumentation 


Aspect 


Reflect and evaluate: Reflect on and evaluate the content of a text 


Item format 


Open constructed response 



Extending the descriptive power of the PISA scales by manipulating item difficulty 

In PISA 2000, PISA 2003 and PISA 2006 it has been noted that, while the level of proficiency of students can 
be located accurately, there is a shortage of descriptive information about what students at the extremes - 
particularly at the lower end of the distribution - know and can do as readers. This is because there are few 
existing PISA tasks at the very easy end and the difficult end, where the proficiency level of significant numbers 
of students in all participating countries is located. In developing tasks for PISA 2009, therefore, there was an 
emphasis on including some very easy and some very difficult items. In addition to enhancing the descriptive 
power of the scale, better matching of the item difficulties to the student achievement distributions in each 
country may improve the reliability of the population parameter estimates. Moreover, the test experience for 
individual students, particularly those performing at very low levels, will become more satisfying. 

Developing items for the lower levels of proficiency was achieved by manipulating elements from PISA's 
descriptive framework as follows: 

■ using shorter and simpler texts 

■ ensuring closer literal match of terms between the item and the text 

■ providing more direction about the location in the text of the information relevant for solving the item 

■ addressing personal and familiar experiences in reflecting on and evaluating content items, rather than 
remote, abstract issues 

■ addressing concrete features in reflecting on and evaluating form items 

The following two examples illustrate tasks designed explicitly to address the extremes of the difficulty 
continuum in PISA reading. The first is a very easy item, the second a very difficult one. These two items were 
administered in the PISA 2009 field trial. 
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PRINT READING UNIT 4 



SUPERMARKET NOTICE 




/ 

/ 

/ 



Peanut Allergy Alert 

Lemon Cream Biscuits 

Date of alert: 04 February 
Manufacturer's Name: Fine Foods Ltd 
Product Information: 1 25g Lemon Cream 
Biscuits (Best before 1 8 June and Best before 
01 July) 

Details: Some biscuits in these batches may 
contain pieces of peanut, which are not included 
in the ingredient list. People with an allergy to 
peanuts should not eat these biscuits. 

Consumer action: If you have bought these 
biscuits you may return the product to the place 
of purchase for a full refund. 

Or call 1 800 034 241 for further information. 



/ 



This public notice consists of a very short text that has an everyday function: to warn about the possible danger 
of a product to consumers and to give advice to return the product for a refund. While the formatting of the 
stimulus reflects the international standard for product recall notices, many students may not have seen this 
kind of notice. Nevertheless, the content of the warning is clearly set out and a minimum number of words is 
used. Lemon biscuits were chosen as the product because of their familiarity and likely appeal. As mentioned 
above, in developing very easy items, short simple pieces of stimulus with familiar content were sought. This 
was not only to make the cognitive load of the items lighter, but also to present texts that were unlikely to 
intimidate students with low reading proficiency, since such readers can easily be discouraged from even 
attempting to read something that they believe looks too hard or too long. The text format classification of the 
supermarket notice is non-continuous, as it consists of a list of described features. In terms of text type, the 
notice is instructional: it provides directions on what to do if you have bought the product. 
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Of the five questions addressing this stimulus that were administered in the field trial, four were successfully 
completed by more than 80% of students. The one reproduced below was the easiest of all, with well over 90% 
of students gaining full credit. Further questions can be found in Annex At . The numbering of the questions is 
identical to the numbering used in the test booklets given to students. 



Question 3: SUPERMARKET NOTICE 

Whgt is the ngme of the company thgt mg4e the biscuitsf 



To answer this question successfully the student needs to locate a single explicitly stated piece of information in 
the text, using a synonymous match between the task direction and the text (company/manufacturer). The fact 
that the whole text is very short, and that the needed information is near the beginning of the text, adds to the 
easiness of the task. The response format for the task is described as a closed constructed response, since only 
one answer (with a small range of variants: Fine Foods or Fine Foods Ltd.) is given full credit. 

Table 1 .1 1 summarises the framework characteristics of the Supermarket Notice question. 



Tgble 1.11 Framework characteristics of sample task: Supermarket Notice 



Situation 


Public 


Text format 


Non-continuous 


Text type 


Instruction 


Aspect 


Access and retrieve: Retrieve information 


Item format 


Closed constructed response 



Example 5, the last of the examples of reading items included in this section, was designed to help elaborate 
the description of student proficiency at the top end of the scale. 
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PRINT READING UNIT 5 



DEMOCRACY IN ATHENS 



Part A 

Thucydides was a historian and military man who lived in the fifth century BC, during the Classical Creek 
period. He was born in Athens. During the Peloponnesian War (431 BC to 404 BC) between Athens and 
Sparta he was in command of a fleet whose mission was to protect the city of Amphipolis in Thrace. He failed 
to reach the city in time. It fell into the hands of Brasidas, the Spartan general, which forced Thucydides 
into a twenty-year exile. This granted him the opportunity of collecting detailed information from the two 
warring factions and the possibility of doing research for his work History of the Peloponnesian War. 

Thucydides is regarded as one of the great historians of Ancient times. He focuses on natural causes and the 
behaviour of each individual rather than on fate or the intervention of divinities to explain the evolution of 
History. In his work, facts are not presented as mere anecdotes; rather, they are explained in an attempt to 
find out the reasons that led the main characters to act as they did. Thucydides' emphasis on the behaviour 
of individuals is why he sometimes introduces fictitious speeches: these help him explain the motivations 
of the historical characters. 

Part B 

Thucydides attributes to Pericles (fifth century BC), the Athenian ruler, the following speech In honour of the 
soldiers who fell in the first year of the Peloponnesian War. 

Our system of government does not copy the laws of neighbouring states; we are rather a pattern to others 
than imitators ourselves. Our system is called democracy, since its administration depends on the many 
instead of the few. Our laws afford equal rights to all in their private affairs, whereas the prestige in public 
life depends on merit rather than on social class. 

Social class does not prevent a person from holding any public position either ( ). And, at the same time that 
we do not interfere in private affairs, we do not break the law as regards public matters. We give our obedience 
to those whom we put in positions of authority, and we obey the laws themselves, especially those which are 
for the protection of the oppressed, and those unwritten laws which it is an acknowledged shame to break. 

Furthermore, we provide plenty of means for the pleasure of the mind. The games and sacrifices we celebrate all 
the year round, and the elegance of our private places of residence, form a daily source of pleasure that helps to 
banish any worry; while the many inhabitants of the city draw the produce of the world into Athens, so that to 
the Athenian the fruits of other countries are as familiar as those of his own. 

Democracy In Athens consists of two fairly short but dense texts. The first is classified as expository, although 
the first paragraph if considered alone could better be described as narration, since it gives an account of when 
something happened, referring to a sequence of events in a person's life. However, in the context of the whole 
of Part A, the biographical paragraph serves as an introduction to the more substantial second paragraph, which 
places Thucydides in the context of ideas, describing his originality as an historian. Part A as a whole, then, 
provides an explanation of concepts or mental constructs, which is a marker of expository texts. 

Part B presents a sample of one of the "fictitious speeches" written by Thucydides that are referred to in Part 
A. Part B is an argumentative text, words imagined as having been spoken by Pericles in a speech of political 
persuasion. Part of the challenge of reading the stimulus as a whole is understanding the relationship between 
the two texts: it is not stated explicitly but can be inferred from the last sentence of Part A and the introduction 
to Part B. Other features that make this stimulus likely to be relatively difficult for 1 5-year-olds are its remoteness 
from their everyday experience, the abstractness of the language and the formal register, particularly of the 
rendition of Thucydides' writing. On the other hand it is reasonable to suppose that most students at the end 
of their compulsory schooling are fairly familiar with history as a concept, and have some notion - even if 
not necessarily articulated - of what democracy might be. This assumed background was judged to provide 
sufficient context for students to approach the Democracy In Athens material. 
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The questions related to this stimulus that were administered in the PISA 2009 field trial reflect two of the 
aspects: access and retrieve and integrate and interpret. Some of the questions focus on one or other of the two 
Parts, and some are designed to draw on an integrated understanding of the two. The one shown here draws 
mostly on Part B. Further questions can be found in Annex At . The numbering of the questions is identical to 
the numbering used in the test booklets given to students. 



Question 3: DEMOCRACY IN ATHENS 

One purpose of the speech in Pgrt B w^s to honour soMiers who fell in the first yegr ofthe 
Peloponnesian Wgr. 

Whgt wgs ANOTHER purpose ofthis speechf 



This is is one of the most difficult items administered in the PISA 2009 field trial. It is coded as a partial credit 
item, with full credit typically gained only by the most proficient readers in the field trial sample. To gain full 
credit, the response needs to identify the purpose of Pericles' speech by linking it to its context, which is partly 
provided in the expository text as well as more directly by the speech itself and by its introduction. The full 
credit response therefore needs to be underpinned by a full and detailed understanding of both texts, and to 
use a high level of inference in an unfamiliar context, dealing with some ambiguity (about the authorship of the 
speech). Responses in this category refer to Pericles' political motivation, possibly by mentioning such purposes 
as persuading soldiers to continue the struggle; consoling the families ofthe dead; fostering pride in the citizens 
of Athens; or stressing the virtues of Athens compared to Sparta or other cities. The following are examples of 
full credit responses: 

^ to make people proud of Athens 
o to promote the benefits of the Athenian democracy 

o making people think Athens is still ok, despite the fact that they are in trouble right now 
•> to reinforce positive thinking and positive attitudes 
o to make people aggressive against Spartans 

Alternatively full credit answers could refer to the more abstract level of meaning implied in Part A: Thucydides' 
purpose of understanding Pericles' psychology or way of thinking for example; 

to explain the motivation of Pericles as an historical character. 

Full-credit answers do not need to be long or elaborate; yet just over a quarter of responses in the field trial were 
in this category. About one-fifth of participants in the field trial were able to demonstrate a partial understanding 
of the text, indicating that its substantial content was about democracy but without registering the persuasive 
intent. Responses such as the following were assigned a partial credit: 

o to introduce democracy 
o to explain democracy to the people 
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Building tasks in the electronic medium 

In this section some of the major issues in constructing and operationalising the electronic reading literacy 
assessment are considered: the relationship between navigation and text processing; analysis of tasks with a 
view to controlling for item difficulty; response formats; and some issues around coding and scoring. The section 
ends with a note on the way the students' progress through the electronic reading assessment is controlled. 



Relationship between navigation and text processing in the electronic reading assessment 

Reading electronic texts requires many of the same skills and understanding as reading in the print medium, 
but reading in each medium also requires unique skills and knowledge, particularly about the conventions 
that are part of the reading context. Just as knowing how to turn a page and which way up to hold a book 
are essential pieces of knowledge for print readers, knowledge of some techniques of navigation and some 
navigation features are part of being literate in the electronic medium. Such skills and knowledge should be 
regarded as "ICT skills" that are in conjunction with reading literacy. Both reading of text, as it is conventionally 
understood, and the ability to navigate within the electronic medium are conceived of as integral to proficiency 
in electronic reading. 

All tasks in the electronic reading assessment require students to read, but individual tasks require varying 
proportions of text processing (for example, interpreting a paragraph of prose) and navigation. The relationship 
within electronic reading tasks between the mental processing devoted to navigation decisions, and that 
devoted to textual processing, is represented in Figure 1 .7 below. 



Figure 17 ■ Relationship between navigation and text processing in electronic reading tasks 



i 




Text processing 



Any individual task should be imagined as occupying a space somewhere within the field of the graph. 
The horizontal axis represents the cognitive load that is imposed by processing the text and the vertical axis 
represents the cognitive load imposed by the navigation. In any given reading task in the electronic medium, 
there will be more or less loading on the navigation and on the text processing elements. Some tasks will be 
very straightforward and require little of either. This kind of task would appear at the bottom left of the graph, 
near the origin. A second kind of task will require the reader to deal with a text that is immediately fully visible 
to the reader on the screen; therefore little or no navigation is required. This kind of task, with a heavy loading 
on text processing and light loading on navigation, would appear along the lower shaded area of the graph. 
A third type of task might demand one or more steps in navigation across a number of sites, with just a few 
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words on each page. This kind of task, requiring mainly navigation and minimal negotiation of text, would 
be located somewhere on the shaded area of the graph along the vertical axis. Finally, a fourth type of task, 
involving substantial navigation and the necessity to process a substantial piece of text (in terms of quantity and/ 
or complexity) would be mapped in the unshaded area of the graph, indicating significant loading in terms of 
both navigation and text processing. The further from the origin of the graph, the more difficult the task. 

As the unshaded area of Figure 1 .7 indicates, most electronic reading tasks load significantly on both navigation 
and text processing proficiency. A relatively small number of tasks do not depend much, if at all, on navigation, 
but focus on straight text processing. It is considered necessary to include some tasks reflecting this type of 
reading in the electronic reading assessment since they represent one kind of real-life reading demand in 
the electronic environment. Failure to include this kind of task would artificially distance what is involved in 
electronic reading from print reading. Conversely, a small percentage of tasks depend mainly on navigation 
processing and require little, if any, text processing. Inclusion of a few such tasks, in which the reader needs to 
move around the electronic environment using a repertoire of knowledge and skills about the conventions of 
the medium and thinking about the relationships between the text objects and the navigation tools, is also seen 
as a valid contribution to a measure of electronic reading. 

Analysis of electronic reading tasks 

In order to capture the complexity of the steps that the reader needs to perform in the process of arriving at 
an explicitly called-for response, a system of analysis was used by the test developers to describe the text 
processing and navigation components of each task. 

For any task with a moderate degree of complexity in the electronic medium, the reader is likely to have 
several possible ways of proceeding. For the purposes of describing and analysing subtasks, the test developers 
imagined an optimally efficient, but comprehensive, sequence of steps, where each step was marked by an 
action. The action could be a click on a specified link or function button; a text response in the browser area or 
the task area; selection from a set of alternatives in a multiple-choice format; or simply scrolling. 

In the analysis, for each subtask completed with an action, the following variables were tabulated: text 
complexity; navigation tool/text used; aspect and description; and action. Each part of the task could therefore 
be described and, potentially, the difficulty of the whole task predicted. An example of an electronic reading 
task from the field trial showing this kind of analysis is provided below. 
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ELECTRONIC READING UNm LET'S SPEAK 

SCREEN 1A 




This unit was based on an online discussion forum on the subject of the challenges of speaking in public. The 
discussion is initiated by Mischa, whose blog entry at the bottom of the discussion forum screen (shown below 
in screen 1e) refers to her terror of speaking in public, to a classroom audience, and asks for help and advice. 
The theme of the discussion, set in an educational situation, is an example of a context that would be familiar to 
most of the PISA students. In terms of text format and text type, it bears some similarity to the Student Opinions 
text illustrated earlier, in that it is categorised as a multiple text, from a number of authors, and argumentative 
in rhetorical structure. However, unlike Student Opinions, Let's Speak presents an interactive situation in which 
the contributors are responding directly to each other. This is a new, or at least much accelerated kind of 
exchange that is an increasingly prevalent form of communication. In this kind of multiple text, understanding 
of each text is partly dependent on following the chain of contributions. 

The discussion forum page is quite long, comprised of eight entries. In order to read the initiating entry, it is 
necessary to scroll down. Screens 1 b to 1 e show what the reader sees when scrolling down. 
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SCREEN IB 




SCREEN 1C 
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SCREEN 1 D 



^ Let^ Speak - &kjcation Network Forums - E022P01 - Internet Browser 



1 ] 



Address http://www.educationnetworkforurTs.(M^ 



□ □ 



]a 



Educatior\ Network Poritfns 



Mark 


January 28 13:28 




Yes 1 agree with everythirtg you say. You can't avoid it. 1 fiourxJ a helpful online article by a 
Doctor Nauckunaite. Take a look. 


Posts: 24 




Lauren 


January 27 13:12 




1 don't think avoiding public speaking is a good idea - it is better to try It. and conquer your 
fear in the process. You can't run away from speaking in public all your life. Even if you are 
very scared of speaking in puUic. th^e are things you can do to overcome your fear. 


Posts: 3 




Tobias 


January IS 16:40 




1 rehearse important speeches at home. 1 read them out loud using the visual aids 1 will use 
when 1 make the speech in pii)lic. That way. r>ot only is my mind less likely to go blank 
when rm talking, but my speech will also be supported by the visuals. It is important that 
vnu dn r¥)t iiist mad vmr srMer.h stmioht frnm vmr notes Ynu need to he ahie to sneak 



E 



E 



SCREEN IE 
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In addition to the starting page, the unit includes only one other piece of stimulus, which is accessed by clicking 
on an embedded link in one of the blogs that recommends it as "expert advice". The second screen, advice from 
Doctor Nauckunaite, also requires some scrolling. 

SCREEN 2A 



Let% Spe^k - Tps on Pubic Speaking - E022P04 • Internet Browser 



Address http://www. unid. lu/phlology/ruuckijnafte/pubic-speaking.htnl 



□ □ 



I Education ^4etwo^k Forums | Tps on Pubic Speaking 




Tips on Public Speaking 

n is natural to be r>er\ous when you have to give a speech. 
Coricentrate. Try r>ot to think about how you appear to others (x how 
r>ervous you are. but only about the subject of your speech. 

People become most nervous when they feel that others can see their 
lack of confider>ce. Knowing how to conceal your sense of fear 
diminishes the fear itself. 

Sir>ce people are most nervous at the beginning of the speech, one 
practical way to overcome fear is to learn the beginning of your 
speech by heart. Before commercing the speech, look around at your 
audience. If you krx)w exactly vi/ho it is you are speaking to. you will 
feel more at ease. 

If you are feelinq overcome by fear during your speech, try not to look 

SCREEN 2B 




B 



Left Speak - Tps oo Pubic SpeaWrvg - E022P04 - Internet Browser [□] 

I I I Address http://www.unid.liVpNk>togy/nauckurHite/pubic-spealcing.htrrt | 1 1 [ | j 



Education l^tetwork Forums I TpsooPi*icSpealdf>g | 

■ vwua yvu oie, uui \/i iiy ola/ui u rc ouujevi ui y\/ui i. 

People become most neneus when they feel that others can see their 
lack of confiderce. Krx>wir>g how to conceal your sense of fear 
diminishes the fear itself. 

Sirce people are most nervous at the beginning of the speech. or>e 
practical way to overcome fear is to learn the beginning of your 
speech by heart. Before commencing the speech, look around at your 
audierce. If you know exactly who it is you are speaking to. you will 
feel more at ease. 

If you are feellr>g overcome by fear during your speech, try not to look 
at a particular audierce member. Instead, direct your gaze toward the 
middle of the audience as a whole. When you use that technique, 
both those sitting in front and those towards the back of the audience 
will feel that you are. in fact, looking at them. Enunciate each word 
clearly. Nothing will soothe you more than your own voice sounding 
clear, and in control. 

Extract from 

Teaching of Oratory, by Or. Z. N.iuckunaitc. Faculty of Philology. 
Vilnius Pedagogical University, Lithuania. 
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This electronic reading unit, which was administered in the field trial for PISA 2009, included several tasks that 
required students to understand the organisation of the website, to identify main ideas both across the blog 
entries and within an individual entry, and to recognise the existence of conflicting opinions. The final task 
directed students to read the last entry (at the top of the discussion forum page) in which Mischa has, in an 
imagined scenario, read all the information provided and is now requesting some final summary advice. This 
task is reproduced below. 

Task: LETS SPEA K 

Look at Mischa's post for March 10. Click on "Write a Reply" and write a reply to Mischa. In your 
reply, answer her question about which writer, in your opinion, knows the most about this issue. 
Give a reason for your answer. 

Click "Post Reply" to add your reply to the forum 

This is a task that requires access and integration of several pieces of information. Mischa's second blog entry 
asks the reader to consider and compare four short texts (those of Julie, Tobias, Psych OL and Dr. Nauckunaite). 
It also requires an evaluation of the contributions, in terms of either their professional credentials, or in terms 
of the intrinsic quality and persuasiveness of the arguments. It is classified as a complex item because it draws 
significantly on all three aspects: access and retrieve, Integrate and interpret and reflect and evaluate. 

An added dimension of the demand of the task is that the student needs to demonstrate some proficiency in handling 
the formal structure and navigational conventions of the message-based environment by scrolling, clicking on a link 
that is embedded in the text, and finally clicking on another link (a button) to write a reply. Once the student has 
clicked on Write a reply , the screen below appears, with an area in which the response can be entered. 

SCREEN 3 




The coding of this item for the PISA 2009 field trial was based on the text response that the student enters in the 
Write a Reply area. (Note that full credit could be obtained for the response without clicking on Post Reply - 
that detail was added in the interest of authenticity.) However, in developing the item, both the text processing 
requirements and the navigational requirements were deliberately manipulated to shape the task for maximum 
contribution in populating the information space of the assessment. Table 1 .12 shows a simplified version of 
how this Let's Speak task can be analysed in terms of its text processing and navigation components. 
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Tgble 112 Analysis of a task from electronic reading assessment, Lets Speak 



step 


start page / 

Required text processing / 
Text complexity rating 


Required 
navigation tools / 
features 


Aspect / text processing description 


Action 


1 


Screen la 

One short argumentative text 
Rating: medium 


Scrollbar 


Interpret: form an understanding of the 
question posed in Misha's message 
of March 10. 

Access: infer that the messages of the four 
entries referred to in Misha's message can be 
accessed by scrolling, with the first blogger's 
name ("Julie") already visible. 


Scroll down 


2 


Screen 1 b 

Two short argumentative texts 
Rating: medium 


Scrollbar 


Retrieve: match on two names in Mischa's 
message ("Julie" and "Psychologist OL"). 
Interpret: form a broad understanding of 
the main ideas expressed in Julie's and in 
Psychologist OL's entries. 

Access: infer that entries of other required 
bloggers are accessible by scrolling 


Scroll down 


3 


Screen 1c 

Two words highlighted in a short 
argumentative text 

Rating: low 


Embedded link 


Access and retrieve: locate 

Dr Nauckunaite's link embedded in Mark's 

blog. 


Click on 

embedded link in 
Mark's blog 


4 


Screen 2a 

Formal text comprising expository 
and instructional elements 

Rating: medium to high 


Scrollbar 


Interpret: form a broad understanding of the 
main ideas expressed in first part of 
Dr Nauckunaite's page. 

Access: infer that article continues below 
bottom of screen 


Scroll down 


5 


Screen 2 b 

Formal text comprising expository 
and instructional elements 

Rating: medium to high 


Back button 


Interpret: form a broad understanding of the 
main ideas expressed in second part of 
Dr Nauckunaite's page. 

Access: return to discussion forum page 
using back button (navigation direction 
provided explicitly in task) 


Click on Back 
button 


6 


Screens 1 a to 1 e 

Eight short argumentative texts (skim) 
Screen 1e 

One of two short argumentative texts 
Rating: medium 


Scrollbar 


Access: infer that further scrolling is required to 
locate the last entry named in Mischa's post. 
Retrieve: match on name in Mischa's 
message ( Tobias ). 

Interpret: form a broad understanding of the 
main idea expressed in Tobias's entry. 


Scroll down 


7 


Screen le 

Write a Reply button 
Rating: very low 


Write a Reply button 


Access: access page to write a reply to 
Mischa 


Click on Write a 
Reply 




Screen 3 








8 


Text box with Write a Reply button 
[recall of 3 short argumentative texts 
from Screens 1 a, 1 b and 1 c and 
formal text comprising expository and 
instructional elements from Screens 2a 
and 2b] 


None 


Reflect and evaluate: generate an evaluation 
of the most authoritative text, combining 
prior knowledge with information from three 
short argumentative texts and one longer 
expository/instructional text 


Text entry 
response 




Rating: very high 








9 

(Optional) 


Screen 3 

Post Reply button 


Post Reply button 


Not applicable 


Click on Post 
Reply 
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For this task, nine distinct steps are described (the last one optional). However, except for step 8, the order of 
the steps could be changed to achieve exactly the same result. For example, step 1 could be followed by step 3; 
or the sequence could begin with step 7 (but using the "Write a Reply" button shown in screen 1 a, and then the 
"Back" button to return to the main page of the forum). There are many other possible variations in the sequence. 
As this task illustrates, even with this relatively restricted set of linked pages, readers in the electronic medium, 
to a degree, construct their own text in terms of the order in which they access and process information. The 
completion of step 8, for full credit, implies good navigation skills in reading electronic text (steps 1 to 7), and 
also strong text processing skills, since the response requires processing, integration and evaluation of multiple 
texts, at least one of which is quite demanding (see steps 4 and 5). 

An analysis of tasks at this level of detail serves at least four functions. First, the analysis assists test developers in 
ensuring that the elements of the framework are reflected in the tasks and reflected in the appropriate proportions. 
Second, it helps in predicting the difficulty of tasks in a systematic fashion so that an appropriate range of item 
difficulties across the instrument is achieved. Third, the analysis is used to underpin the construction of a 
described proficiency scale (or scales) for reporting the results of the electronic reading assessment for PISA 
2009. Fourth, the analysis will potentially contribute to research in the field by articulating what proficiency in 
electronic reading means, based on empirical evidence. 

A fifth function, analytical scoring of text processing and navigational components, remains a potential for 
computer-based assessment that will not be operationalised in the main PISA 2009 analysis. The combination 
of data from the eight steps described in Table 1 .12 could be used to determine a partial credit coding scheme 
for the task, where subscores could be given simply for accessing (for example) the Doctor Nauckunaite page, 
even if the student gained no credit for the final text-entry response. Conversely, one might consider giving full 
credit only to those responses with all eight steps completed (in whatever sequence). 

In its present form and coding, this task could arguably be completed with minimal effort of both text processing 
and navigation (say, by completing only steps 1 , 7 and 8). However, by the same logic, the Student Opinions task 
could be completed by reading only the last sentences of Ana's piece. We are often in the position of speculating 
about how test-takers actually deal with tasks cognitively. Computer-based testing offers the potential to understand 
much more precisely the cognitive processes - indicated by the pathways taken in navigating across the texts - 
that readers engage in. The technology used for the electronic reading assessment in PISA 2009 has the potential 
to provide information about which pages are visited and which navigation tools are used, the sequence of actions 
and the time spent on each item. The data collected from the electronic reading assessment will no doubt provide 
a rich base for research. However, given that computer-based assessment on a large-scale is at an early stage of 
development, scoring for this cycle will be confined to established methods. 

In the electronic reading assessment, items may be coded automatically, or by experts. Considering practicality, 
reliability and (above all) cost, it is preferable to maximise the proportion of items that can be automatically 
coded, and to minimise those that require expert coding. On the other hand, since electronic reading is being 
conceptualised as part of the general reading framework, it is important to maintain some parallels in item format 
between the electronic and print instruments. Accordingly, about 30% of the electronic reading items require 
expert coding. See Table 1 .7 for the distribution of electronic reading tasks by coding requirement and aspect. 

Automatically coded items include multiple-choice items, selection of links from drop down menus, and clicks 
on radio buttons in the browser area. 

Expert-coded items include any item involving a generated response where a degree of judgement is needed 
for coding. The item format for this kind of response in PISA 2009 is a text response, either within a simulated 
environment such as an e-mail message or blog (as in the example from Let's Speak shown above), or in the more 
transparently test-based task response area. Responses that require coding by expert coders include those for 
which students are asked to compare and contrast information across texts, to justify a choice by demonstrating 
that is satisfies prescribed conditions, or to evaluate the credibility of a source. Such items are generally used 
only when there is no satisfactory alternative (that is, no appropriate machine-scored alternative). 

Control of the delivery of tasks in the electronic reading assessment 

As the screen shots for the task from Let's Speak show, the interface for an electronic reading unit (a set of 
items related to common stimulus) has two distinct areas: a task area in the lower part of the screen, where 
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the question or instruction is located, and a browser area in the upper part of the screen, where the stimulus is 
located. The task in the task area remains fixed for the duration of an item while the student can navigate around 
the browser area to access different simulated web pages or applications in the course of completing a task. 

In the electronic reading assessment, both units and items within units are delivered in a fixed order, or 
lockstep fashion. The lockstep procedure means that the students are not able to return to an item or unit 
once they have moved to the next item/unit. Each time students click the Next button on the test, a dialog box 
displays a warning that they are about to move to the next item and will be unable to return to the previous 
item. At this point students can either confirm they want to move on, or cancel the action and return to the item 
they had been viewing. 

An advantage of this approach is that it maximises the independence of items within and across units, since 
students cannot find clues in later tasks that might help them answer earlier ones. Put more positively, later 
items can reveal the answers to earlier items without enabling previous answers to be changed. This feature 
therefore allows optimum use of a stimulus that is expensive to produce. 

A further feature of the task delivery design is that the page that is visible in the browser area at the beginning 
of each item is fixed: that is, every student sees the same page at the beginning of a given item, regardless of 
where they finished the previous item. This feature contributes to item independence by ensuring that students 
are not given any advantage or disadvantage in relation to a new item because of where they happened to be 
at the end of the previous item. This design feature has also been implemented to reduce frustration, both for 
those students who might be lost in a simulated web environment, and for those who would otherwise have 
needed to retrace their steps. 

Although only 1 9 of the PISA participating countries and economies have administered the 2009 international 
option of reading of electronic texts, what we have learnt about the nature of the domain and about the 
operational challenges and opportunities encountered in implementing a large-scale computer-delivered 
assessment will be shared by all OECD countries and partner economies, and will contribute to building more 
refined understandings that can be applied to future PISA assessments. 



MOTIVATIONAL AND BEHAVIOURAL CONSTITUENTS OF READING LITERACY 

Reading engagement and metacognition, which were discussed very briefly in earlier versions of the framework, 
are now, in the PISA 2009 cycle, given a more prominent position and more elaborated discussion, in recognition 
of their importance in relation to reading proficiency. 

Reading engagement 

Why reading engagement matters 

The development of reading literacy is not confined to the development of skills and knowledge. It also involves 
motivation, attitude, and behaviour. Current research recognises these elements as key factors relating to 
reading achievement (see Guthrie & Wigfield, 2000; McKenna, Kear, & Ellsworth, 1995). Empirical studies 
have documented the link between reading practices and reading achievement in both adult and student 
populations (Campbell, Voelkl, & Donahue, 1997; Guthrie & Wigfield, 2000; OECD and STATCAN, 2000). 
Results from PISA 2000 show that, in every participating country, students' levels of reading engagement were 
positively and significantly correlated with their reading proficiencies. In fact, engagement in reading had the 
largest median correlation with achievement, exceeding even the median correlation between reading literacy 
and socio-economic status (OECD, 2002). Collectively, these and other findings point to the important role 
that engagement in reading can play in reducing some of the gaps seen in performance between the various 
subgroups in each participating country. They argue for the inclusion and even expansion of such a measure 
in PISA 2009. 

The PISA 2009 framework for reading literacy includes the reading of electronic texts as an important part of 
the construct. To address the goal of measuring engagement in both paper and electronic reading, a unified 
framework is proposed. This framework includes two constructs: individual engagement and educational context. 
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The foundations of the concept of engagement lie in the self-determination theory (Ryan & Deci, 2000). This 
theory proposes that individuals are most well developed when they are self-determining. In this state they 
possess values and goals that are in tune with their culture while still being competent and confident in directing 
their own actions. A self-determining reader is intrinsically motivated, which refers to reading for its own sake 
and value. He or she reads widely for a variety of interests and purposes, and takes ownership of favourite topics 
or authors. A self-determining reader holds values, beliefs and goals for reading that enable him or her to pursue 
educational, occupational, personal and societal aims and activities. 



Defining and measuring individual reading engagement in PISA 

The PISA 2009 definition of individual reading engagement is as follows: 

Individual Reading Engagement refers to the motivational attributes and behavioural characteristics of 
students' reading. 

Prominent constructs in the empirical literature on individual reading engagement include interest, perceived 
autonomy and social interaction. The behavioural characteristics include the amount and breadth of reading 
activities. 

Current research suggests that engaged readers possess well-formed interests and favourite topics or types 
of reading material (interest); they value being in control of their reading, and self-initiate reading activities 
(autonomy); they rely on a social network to extend their competencies and share their knowledge and 
experience (social disposition); and they read frequently and widely (behaviour). 

In PISA 2009, these four characteristics (each broadly interpreted) are operationalised: 

■ interest in reading - disposition to read literature and information texts for enjoyment and the satisfaction of 
curiosity 

■ perce/vec/ autonomy -perceived control and self-direction of one's reading activities, choices, and behaviours 

■ social interaction - social goals for reading and interactive competence 

■ reading practices - behavioural engagement referring to the amount and types of reading activities 

Specifically, reading practices are defined as the self-reported frequencies of participating in reading activities 
with diverse content in various media. Importantly, readers may engage with different types of texts in different 
ways. They might read to gain knowledge or information; for literary experience; to perform a specific task such 
as retrieving a piece of information; or for social communication. While these experiences are not necessarily 
mutually exclusive, they can reflect real differences in how students might interact with a particular text and 
whether they engage with either print or electronic texts. For example, very few people acknowledge "cuddling 
up" with a laptop to read fiction - they much prefer to use printed materials for this type of reading. On the 
other hand, many people report using computer technologies for social purposes. They place a high priority 
on e-mail, text messaging, chat rooms, correspondence, list servers, blogs and community bulletin boards. 
Electronic texts seem to be used largely for social purposes. 

At the lowest levels of reading engagement, as defined in PISA, students spend little time reading for pleasure 
or interest, read a narrow range of texts, and have little motivation to read either independently or in a social 
context. 

Highly engaged readers, on the other hand, spend substantial amounts of time reading for enjoyment. They read 
a wide variety of texts in both print and electronic media, though the particular kinds of texts they read will 
typically vary according to gender and nationality (OECD, 2002). Such individuals not only consider reading to 
be valuable and interesting by nature; they also acknowledge the significant role it plays in their social relations. 



The importance of educational context for reading engagement 

As PISA 2009 is intended to inform educational policy, it is important to include variables that are responsive 
to policy decisions. One important policy-relevant variable is the classroom and school context for reading 
engagement. 
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Development of self-determination in reading evolves when individuals enjoy support for the values and 
behaviours of reading from significant others: people who are important to them. Initially, the parents convey the 
value of reading to the developing child. Reading is externally guided. Gradually, individuals internalise the goals 
and values that have been communicated to them by others and become convinced that reading is valuable. 
They may identify reading as being beneficial to their schooling and their future participation in society, although 
reading may not be a favourite pastime. Eventually, reading attains the status of having personal significance and 
becomes a preferred source of mental stimulation and emotional satisfaction. 

The pathway to self-determination in reading for students depends on support from significant others. A 
favourable context assures competence in reading and expands autonomy in directing one's own reading 
activities. When the family and school context give the individual a sense of confidence (perceived competence) 
and autonomy (being in charge of one's self) in reading, the individual grows towards intrinsically motivated 
and self-determined reading. 

The teacher is a "significant other" for reading literacy. A host of studies show that teachers who improve students' 
sense of ownership and competence enable students to become active readers who are high achievers in reading. 
By contrast, teachers who neglect these instructional practices undermine students' efforts to become self- 
directing, resulting in students who are disengaged from reading and fail to progress in reading achievement 
(Guthrie, 2008). 

In summary, the educational context of instructional practices significantly determines students' levels of 
reading engagement. Given the demonstrated relation between engagement and achievement, the collection 
of information about classroom support for reading engagement is therefore of significant policy relevance. 

Defining and measuring the educational context for reading engagement in PISA 

The PISA 2009 definition of classroom support for reading engagement is as follows: 

Classroom Reading Engagement refers to the students' perceptions of support from the teacher, classroom 
and school for the motivational attributes and behavioural characteristics of their reading. 

Two characteristics of classroom reading engagement that are operationalised in the PISA 2009 survey are 
relevance and autonomy support. These characteristics are aligned respectively with the characteristics of 
interest in reading and autonomy in the construct of individual reading engagement. 

Relevance. Interest in reading is facilitated by classroom and school contexts that emphasise the relevance of 
texts to student background knowledge and experience (Assor, Kaplan, & Roth, 2002). When students read 
material that is presented as directly related to their personal interests, their comprehension is higher than if 
their reading is driven by a desire for test scores only (Vansteenkiste, Lens&Deci, 2006). Reading materials that 
are linked to immediate experience or laboratory activities are understood better than materials that are not as 
relevant (Guthrie, et al., 2006). Likewise, texts rated as "interesting" are read more thoroughly than other texts 
(Schiefele, 1999). 

Autonomy support. Perceived autonomy, which is the major element of intrinsic motivation, is increased by 
classroom opportunities for choice and control (Skinner, Furrer, Marchand, & Kindermann, 2008). Teachers 
have a wide range of possible ways to share control with students (Flowerday & Schraw, 2000). When teachers 
are trained to share control, such as giving text choice or inviting input into decisions, students' classroom 
engagement increases (Reeve, 2004). 

Collecting information about reading engagement 

Items contributing to the measurement of individual reading engagement are distributed amongst questions 
relating to reading motivation (interest, autonomy, and social interaction) and to reading practices in both 
print and electronic media. Items contributing to the measurement of the educational context for reading 
engagement are framed in the context of questions about classroom activities. 

Items measuring reading engagement appear in the Student Questionnaire, and the construct is therefore also 
referred to in the Questionnaire Framework (Chapter 4). The items are listed in Annex B. They are in the form 
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of general questions in which students report on their behaviour in described contexts. A Likert scale format 
(Agree/Disagree) or a question about frequency of occurrence (Never/Several times a week) is used to collect 
student responses. 

Metacognition in reading 

Why metacognition matters 

Like engagement, metacognition has both a significant correlation with reading proficiency and is responsive 
to teaching and learning. A number of studies have found an association between reading proficiency and 
metacognition (Artelt, Schiefele, & Schneider, 2001 ; Brown, Palincsar, & Armbruster, 2004). Explicit or formal 
instruction of these strategies is believed to lead to an improvement in text understanding and information use. 
More specifically, it is assumed that the reader becomes independent of the teacher after these text processing 
strategies have been acquired and are applied without much effort. By using these strategies, the reader can 
effectively interact with the text by conceiving reading as a problem-solving task that requires the use of strategic 
thinking, and by thinking strategically about solving reading comprehension problems. 

The general finding of the report of the U.S. National Reading Panel (2000) was that remediating poor 
reading literacy is possible through explicit teaching of metacognitive skills. That is, when readers are given 
cognitive and metacognitive strategy instruction, they make more significant gains on measures of reading 
comprehension than students only trained with conventional instruction procedures (Pressley, Graham, & 
Harris, 2006; Pressley, etal., 1989; Rosenshine & Meister, 1994; Rosenshine, Meister, & Chapman, 1996). 

Gathering information in PISA 2009 on those aspects of metacognition that have a demonstrated association 
with reading proficiency can provide the kind of information used for improving reading literacy and therefore 
meet one of PISA's aims: to provide policy makers with strategies for improving the educational outcomes of 
their students. 

Defining metacognition in reading 

Metacognition in reading refers to the awareness of and ability to use a variety of appropriate strategies when 
processing texts in a goal oriented manner. Learning from texts requires the reader to take an active role in their 
reading by making inferences, filling in gaps, and generating macrostructures (conceptualisations of the large- 
scale structure of a text) and elaborations. Engagement in such strategic activities implies an awareness of text 
structure and how it facilitates understanding. It is important to use both knowledge about language and texts 
and topic knowledge in a strategic way in order to: identify relevant information; selectively reinstate previous 
text information; retrieve or reinstate information from long term memory; or perform all three tasks (Baker & 
Brown, 1984; Borkowski & Turner, 1990; Korkel & Schneider, 1992). It also involves an understanding of the 
different processing demands associated with various kinds of tasks, and how to apply this understanding. 

Overall, cognitive and metacognitive knowledge concerning the use of cognitive strategies in general and 
reading strategies in particular can be defined as mental or behavioural activities that help the learner to achieve 
cognitive purposes. Typically, these strategies are effort-consuming, potentially conscious, and controllable 
(Elavell, Miller, & Miller, 1993). Eor example, a reader may be taught to generate questions about a text as 
it is read. These questions are of the why, what, how, when, or where variety. By generating such questions 
and trying to answer them, the reader processes the text more actively. Other strategies relevant to different 
purposes of reading are various forms of highlighting and summing up important text information (identifying 
main ideas); frequent comprehension monitoring and self-checking; and a repertoire of approaches for dealing 
with text difficulties (clarifying). 

Measuring metacognition in reading in PISA 

The concept of reading literacy in PISA encompasses a wide variety of texts, contexts and approaches to 
reading that aim to give broad coverage to the many ways in which reading is important to individuals and 
societies in the 2H* century. The focus of the metacognition construct and the items proposed for inclusion in 
PISA 2009 is specifically on reading to learn - that is, on reading in the educational situation (See educational 
texts under Situation). 
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A person who intelligently uses a particular strategy should have some metacognitive knowledge of that 
strategy, and a person who does not use the strategy is expected to be less knowledgeable. In other words, there 
is a correlation between appropriate pieces of metacognitive knowledge and the effective use of strategies. 
Metacognitive knowledge is a prerequisite for reflective and strategic learning. Not surprisingly therefore, 
consistent and substantial relationships among metacognitive knowledge, strategic behaviour in a text 
processing task, and text recall have been found (e.g. Baker & Brown, 1984; Borkowski & Turner, 1990; Korkel 
& Schneider, 1 992). Nevertheless, the relationship between metacognitive knowledge and the understanding of 
text in a given situation is moderated by students' actual motivation to read or to invest effort. 

There is empirical evidence to suggest that there is a correlation between appropriate pieces of 
metacognitive knowledge and the effective use of related strategies on the one hand, and proficiency 
in reading on the other. An instrument measuring metacognitive knowledge about text comprehension 
was administered to students who took part in the PISA 2000 assessment in Germany. A correlation of 
r = 0.51 between the combined reading literacy scale and the measure of students' metacognitive 
knowledge was found (Artelt, Schiefele, & Schneider, 2001). Research based on PISA 2003, where such 
an approach was also implemented, revealed a similar correlation between metacognitive knowledge and 
reading literacy (Schlagmuller & Schneider, 2006). A similar measurement instrument is being administered 
in PISA 2009. 

Collecting information about metacognition 

In the PISA 2009 assessment, a number of reading scenarios (short vignettes) are presented to students. For each 
scenario, students are asked to evaluate the quality and usefulness of different reading and text comprehension 
strategies for reaching the intended goal. 

The rank order of strategies for each scenario is compared with an "optimal" rank order developed by experts in 
the field of text processing (reading researchers, teachers and educational psychologists). The correspondence 
between the rankings of experts and students is reflected in a metacognition score indicating the degree to which 
students are aware of the best ways of storing text information and understanding memory and comprehension 
goals. In order to achieve high scores on the metacognition test, students must activate knowledge about 
cognitive resources, the nature of the task, and strategies that facilitate understanding, remembering and 
recalling of information. 
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An example of a metacognition task that was administered in the field trial for PISA 2009 is provided below. 

Q Reading t^slc You wgnt to help g 12-yegi'-ol4 stu4ent to un4etstgn4 3 three-pgge text 

gbout gnimgis gn4 plants of the forest. 



How 4o you rgte the usefuhess of the following strategies for helping the 12-yegr ol4 
stu4ent to un4erstgn4 the three-pgge textf 







Score 






Not useful at 
all 






Very useful 




Possible strategy 


(1) 


(2) 


(3) 


(4) 


(5) 


(6) 


a) 


First the 1 2-year-old student writes 
a summary of the text. After that we 
check together whether the summary 
covers the most important points 


□ , 








□s 


□. 


b) 


1 ask the 12-year-old student to read 
the text out loud twice, and then to 
copy it out 


□ , 




□. 




□s 




c) 


After the 12-year-old student has 
read the text aloud, we discuss 
difficult words that he did not 
understand 


□ , 


□ , 


□. 








d) 


1 provide a second text about the 
same topic which we read together 
immediately after reading the first 
one. 


□ , 




□. 






□. 


e) 


1 read the text aloud while the 
12-year-old student underlines 
words he doesn't understand. 1 then 
try to help him clear up what he 
doesn't understand. Then he writes a 
summary. 


□ , 




□. 


n. 




n. 


f) 


The 12-year-old student reads 
the text aloud and 1 correct him 
whenever he makes a mistake. Then 
1 explain the meaning of the words 
that he did not read correctly. 


□ , 




□. 






□. 



Expert raters judge strategies a), c) and e) to be more effective than strategies b), d) and f) in helping a younger 
student to understand information in a text book. Accordingly, the more closely a student approximated the 
experts' ratings of the strategies in this item - a), c), e) > b) ,d), f) - the higher the student scored. For example, if 
a student rated the six strategies respectively as a)6; b)3; c)5; d)3; e)6 and f)1 , that student would receive a high 
score for the item. Conversely a student who, for example, gave the highest rating to strategy f) would receive a 
low score for this item. In the field trial, performance on this item showed a strong correlation with performance 
in the cognitive reading assessment. In other words, students who achieved a high score on this metacognition 
item tended to do well overall on the PISA reading assessment. 
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REPORTING PROFICIENCY IN READING 

PISA reports in terms of proficiency scales which are interpretable in policy terms. In PISA 2000, when reading 
was the major domain, the results of the reading literacy assessment were first summarised on a single composite 
reading literacy scale having a mean of 500 and a standard deviation of 100. In addition to the composite 
scale, student performance was also represented on five subscales: three process (aspect) subscales (retrieving 
information, interpreting texts and reflection and evaluation) and two text format subscales (continuous and 
non-continuous) (OECD, 2001, 2002). These five subscales make it possible to compare mean scores and 
distributions among subgroups and countries by various components of the reading literacy construct. Although 
there is a high correlation between these subscales, reporting results on each subscale reveals interesting 
interactions among the participating countries. Where such interactions occur, they can be examined and 
linked to the curriculum and teaching methodology used. In some countries, the important question may be 
how to teach the current curriculum better. In others, the question may not only be how to teach but also what 
to teach. 

In both PISA 2003 and 2006, when reading was a minor domain, and fewer reading items were administered 
to participating students, a single reading literacy trend scale was reported based upon the overall composite 
scale (OECD, 2004). In PISA 2009, reading is again the major domain, as it was in 2000. A reporting scheme 
using a number of subscales as well as a composite scale can therefore be anticipated. 

Interpreting and using the data 

The reading literacy tasks are constructed and administered to nationally representative samples of 15-year-old 
students in participating countries to ensure that the assessment provides the broadest possible coverage of the 
domain as defined here. However, no individual student can be expected to respond to the entire set of tasks. 
Accordingly, the survey is designed to give each student participating in the study a subset of the total pool of 
tasks, while at the same time ensuring that each of the tasks is administered to nationally representative samples 
of students. 

Reading literacy tasks are arranged along a scale that indicates progressively the level of difficulty for students 
and the level of skill required to answer each item correctly. The scale summarises both the proficiency of a 
person in terms of his or her ability and the complexity of an item in terms of its difficulty. 

Reading literacy tasks used in PISA vary widely in situation, text format, and task requirements, and they also 
vary in difficulty. This range is captured through what is known as an item map. The item map provides a visual 
representation of the reading literacy skills demonstrated by students at different points along the scale. The map 
contains a brief description of a selected number of released assessment tasks along with their scale values. 
These descriptions take into consideration the specific skills that the item is designed to assess and, in the case 
of open-ended tasks, the criteria used for judging the item correct. An examination of the descriptions provides 
some insight into the range of processes required of students and the proficiencies they need to demonstrate at 
various points along the reading literacy scale. An item map (or maps) will be built to illustrate what progress 
means along the scale (or scales) developed for PISA 2009. 

Levels of reading literacy proficiency 

Just as students within each country are sampled to represent the national population of 1 5-year-old students, 
each reading literacy task represents a class of tasks from the reading literacy domain. Tasks at the lower end of 
the reading scale and subscales differ from those at the higher end. Difficulty is in part determined by the length, 
structure and complexity of the text itself. However, while the structure of a text contributes to the difficulty 
of an item, what the reader has to do with that text, as defined by the question or instruction, interacts with 
the text and affects the overall difficulty. A number of variables that can influence the difficulty of any reading 
literacy task have been identified, including the complexity and sophistication of the mental processes integral 
to the aspect of the task (retrieving, interpreting or reflecting), the amount of information to be assimilated by 
the reader and the familiarity or specificity of the knowledge that the reader must draw on both from within and 
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from outside the text. In an attempt to capture this progression of complexity and difficulty in PISA 2000, the 
composite reading literacy scale and each of the subscales were divided into five levels: 



Level 


Score points on the PISA scale 


5 


More than 625 


4 


553 to 625 


3 


481 to 552 


2 


408 to 480 


1 


335 to 407 


Below level 1 


Less than 335 



These levels appear to be a useful way to explore the progression of reading literacy demands within the 
composite scale and each subscale. 

The scale summarises both the proficiency of a person in terms of his or her ability and the complexity of an 
item in terms of its difficulty. The mapping of students and items on one scale represents the idea that students 
are more likely to be able to successfully complete tasks mapped at the same level on the scale (or lower), and 
less likely to be able to successfully complete tasks mapped at a higher level on the scale. 

It is expected that these levels as they were defined for PISA 2000 will be kept for the composite scale used to 
measure trends. For PISA 2009, newly constructed items will help to improve descriptions of the existing levels 
of performance and, ideally, furnish descriptions of levels of performance above and below those established 
in PISA 2000. 

Given the top of the reading literacy scale currently has no bounds, there is arguably some uncertainty about 
the upper limits of proficiency of extremely high performing students. However such students are likely to 
be capable of performing tasks characterised by the highest level of proficiency. For students who are at the 
bottom end of the reading literacy scale there is a greater issue. Although it is possible to measure the reading 
proficiency of students performing below Level 1, at this stage their proficiency cannot be described. Level 1 
begins at 335, yet a certain and significant percentage of students in each country is estimated to be below 
this point on the scale. In developing new material for PISA 2009 an effort has been made to design items that 
measure reading skills and understandings located below the current Level 1 . The intention will be to describe 
what those skills and understandings are, and possibly to define one or more levels below Level 1 . (See the first 
two paragraphs under Extending the descriptive power of the PISA scales by manipulating Item difficulty). 

Reporting PISA 2009 reading literacy 

The new framework for reading literacy extends the domain to include both printed and electronic texts. 
Reporting therefore becomes a little more complicated. It will be important to maintain both the original 
composite scale and subscales to measure trends based on printed texts. These will be the same scales that were 
used to report PISA 2000 data and they will be relevant to all participating countries. 

Table 1.13 below shows the proposed scales and subscales for print reading that will provide trend data for PISA 
2000, 2003, 2006 and 2009. The cells are numbered 1 to 5 to facilitate discussion. Note that each task will be 
classified both by aspect and by text format. 
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TgbleHS Reporting results on print reading instruments 


Aspect 


Access and retrieve 

(previously called Retrieving information) 


1 




Integrate and interpret 
(previously called Interpreting texts) 


2 




Reflect and evaluate 

(previously called Reflection and evaluation) 


3 


Text format 


Continuous 


4 




Non-continuous 


5 



The cells numbered 1 to 3 represent groups of tasks categorised by aspect. These three cells together include all 
the print reading tasks. A composite scale (1 +2+3) will be built to describe performance on the full set of print 
items, and subscales will be extracted based on the tasks for each of the aspects (separate subscales based on 
1 , 2 and 3 respectively). 

The cells numbered 4 and 5 represent tasks classified as continuous and non-continuous respectively. Subscales 
will be extracted from the composite scale (1+2+3) based on these two text formats. A small number of print 
reading tasks will be classified as mixed or multiple by text format; however there are too few of these tasks to 
constitute separate subscales (see Table 1 .2). Mixed- and multiple-text tasks will therefore not contribute to the 
text-format subscales, though they will contribute to the print composite scale and aspect subscales. 

The composite scale based on 1 +2+3, and the subscales 1 , 2, 3, 4 and 5 will be statistically equated with scales 
and subscales developed for PISA 2000, and will therefore form the basis for reporting trends. Though they will 
have different names, the aspect subscales will be directly comparable with the aspect subscales reported in 
PISA 2000. 

For those countries that have chosen to implement the assessment of electronic reading an additional scale 
based only on electronic reading tasks will be created, therefore starting a new trend line. In addition, the 
possibility of constructing an overall PISA reading literacy scale combining information from the print and 
electronic reading assessments will be investigated through analysis of the main survey data. Should it be 
demonstrated that it is valid to combine data from the print and electronic reading assessments, the possibility 
of constructing aspect subscales across the media will also be considered within countries. It may become 
apparent, however, that performances on print and ERA tasks are not highly enough correlated to support the 
construction of an overall scale and subscales. This kind of result will also be of great interest to participating 
countries, suggesting further investigation of which elements of reading in the two media draw on different 
understandings and knowledge, and perhaps pointing to differential levels of proficiency among subgroups (for 
example, males and females). 

Table 1 .14 represents the components of print and electronic reading assessments that might be combined in 
different ways to form scales and subscales. 



Tgble 114 Reporting results across print and electronic reading instruments 



Aspect 


Print reading 


Electronic reading 


Access and retrieve 


1 


6 


Integrate and interpret 


2 


7 


Reflect and evaluate 


3 


8 


Complex 




9 
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The cells numbered 1 to 3 represent the same sets of items as those numbered 1 to 3 in Table 1 .13. Cells 6 to 
8 represent the set of electronic reading items that can be classified according to a single aspect: access and 
retrieve, integrate and interpret, or reflect and evaluate. The cell numbered 9 represents complex electronic 
reading tasks: those that combine all three aspects inextricably. 

As well as the composite print scale and the five print subscales described above, for countries participating in the 
electronic reading international option, a scale based on tasks from cells 6 to 9 will be constructed. In addition, the 
following will be investigated: a composite reading scale across print and electronic media (1 +2+3+6 +7+8+9); a 
composite access and retrieve subscale (1 +6); a composite integrate and interpret scale (2+7) and a composite reflect 
and evaluate scale (3+8). The construction of the cross-media scale and subscales will depend on empirical support 
for the theory that there is a high correlation between proficiency in print reading and electronic reading, and within 
each aspect of reading across the two media. 

Separate subscales for electronic reading independent of print reading will not be constructed, as there will be 
too few Items In the elecronic reading assessment instrument on which to base meaningful subscale reporting. 



CONCLUSION 

An essential concern of PISA is to provide information to policy makers about trends over time. In light of that, 
the analysis of trends must be given priority in any reporting plans, and this will be ensured in PISA 2009 by the 
construction of a scale and subscales that are based entirely on print reading tasks. A different set of scales will 
be built to report on the electronic reading assessment and, where possible, to report the combined results of 
print and electronic reading assessments, therefore providing the basis for establishing new trend lines for future 
cycles. In anticipating a range of options for reporting, the PISA reading literacy framework and assessment are 
designed to provide an array of data capable of richly informing the work of policy makers, educators, and 
researchers. 

The construct of reading literacy that has been described in this document preserves many of the principles 
and operational features of PISA 2000, while also introducing new perspectives on reading literacy. An 
important addition for PISA 2009 is the inclusion of motivational and behavioural components in the definition 
and description of reading literacy. The notion of reading literacy in PISA therefore goes beyond the simple 
measurement of a student's capacity to decode and understand literal information. Reading literacy in PISA also 
involves understanding, using, reflecting on and engaging with written texts, both to achieve personal goals and 
to participate actively in society. 
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Notes 

1 . The discussion in this section refers to reading in both the print and electronic media, unless otherwise stated. 

2. This does not preclude the use of several texts in a single task, but each of the texts should itself be coherent. 
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This chapter begins with PISA's definition of mathematics and 
explains the theory underlying the development of the assessment. 
The PISA mathematics assessment is organised into many different 
categories: situations or contexts; mathematical content and 
mathematical competencies. It also includes four overarching 
ideas: space and shape; change and relationships; quantity and 
uncertainty. The chapter describes the processes and competencies 
needed to solve PISA mathematics questions and explains the 
categorisation of the three competency clusters: reproduction, 
connections and reflections. It uses sample tasks from the PISA 
assessment to further illustrate the framework and then discusses 
how proficiency in mathematics is measured for the analysis and 
reporting of results. 
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INTRODUCTION 

This framework describes and illustrates the PISA mathematics assessment. The term mathematical literacy is 
used to highlight that the PISA mathematics assessment is concerned with the reproduction of mathematical 
knowledge and in addition, in solving the PISA assessment tasks, students are typically required to extrapolate 
from what they have learned in school and to apply mathematical knowledge to authentic problems situated 
in a variety of contexts. The mathematical processes needed to do this, which are based on mathematical 
knowledge and skills, are referred to as cognitive mathematical competencies. The major components of the 
mathematics framework, consistent with the other PISA frameworks, include contexts for the use of mathematics, 
mathematical content and mathematical processes, each of which flow directly out of the definition of the 
domain. The discussions of context and content in this chapter emphasise features of the problems that confront 
students as citizens, while the discussions of processes emphasise the mathematical knowledge and skills that 
students employ to solve those problems. These processes have been grouped into three clusters to facilitate 
a rational treatment of the way complex cognitive processes are addressed within a structured assessment 
programme. 



DEFINITION OF THE DOMAIN 

The PISA mathematics domain is concerned with the ability of students to analyse, reason and communicate 
ideas effectively as they pose, formulate, solve and interpret mathematical problems in a variety of situations. 
The PISA mathematics assessment focuses on real-world problems, moving beyond the kinds of situations and 
problems typically encountered in school classrooms. In real-world settings, citizens routinely face situations 
in which the use of quantitative or spatial reasoning or other cognitive mathematical competencies would 
help clarify, formulate or solve a problem. Such situations include shopping, travelling, cooking, dealing with 
personal finances, judging political issues, etc. Such uses of mathematics are based on the skills learned and 
practised through the kinds of problems that typically appear in school textbooks and classrooms. However, 
they also demand the ability to apply those skills in a less structured context, where the directions are not so 
clear, and where the student must make decisions about what knowledge may be relevant and how it might be 
usefully applied. 

Citizens in every country are increasingly confronted with a myriad of tasks involving quantitative, spatial, 
probabilistic and other mathematical concepts. For example, media outlets (newspapers, magazines, television 
and the Internet) are filled with information in the form of tables, charts and graphs about subjects such as 
weather, climate change, economics, population growth, medicine and sports, to name a few. Citizens are also 
confronted with the need to read forms, interpret bus and train timetables, successfully carry out transactions 
involving money, determine the best buy at the market, and so on. The PISA mathematics assessment focuses 
on the capacity of 1 5-year-old students (the age when many students are completing their formal compulsory 
mathematics learning) to use their mathematical knowledge and understanding to help make sense of these 
issues and carry out the resulting tasks. 

PISA defines mathematical literacy as: 

...an individual's capacity to identify and understand the role that mathematics plays in the world, to make 
well-founded judgments and to use and engage with mathematics in ways that meet the needs of that 
individual's life as a constructive, concerned and reflective citizen. 

Some explanatory remarks may help to further clarify this domain definition: 

■ The term mathematical literacy emphasises mathematical knowledge put to functional use in a multitude of 
different situations in varied, reflective and insight-based ways. Of course, for such use to be possible and 
viable, many fundamental mathematical knowledge and skills are needed. Literacy in the linguistic sense 
presupposes, but cannot be reduced to, a rich vocabulary and substantial knowledge of grammatical rules, 
phonetics, orthography, etc. To communicate, humans combine these elements in creative ways in response 
to each real-world situation encountered. In the same way, mathematical literacy presupposes, but cannot 
be reduced to, knowledge of mathematical terminology, facts and procedures, as well as skills in performing 
certain operations and carrying out certain methods. It involves the creative combination of these elements 
in response to the demands imposed by external situations. 
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■ The term the world refers to the natural, social and cultural setting in which the individual lives. As 
Freudenthal (1983) states: Our mathematical concepts, structures, ideas have been invented as tools to 
organise the phenomena of the physical, social and mental world" (p. ix). 

■ The phrase to use and engage with is meant to cover the usage of mathematics and solving mathematical 
problems, and also implies a broader personal involvement through communicating, relating to, assessing, 
and even appreciating and enjoying mathematics. Thus, the definition of mathematical literacy encompasses 
the functional use of mathematics, in a narrow sense, as well as preparedness for further study and the 
aesthetic and recreational elements of mathematics. 

■ The phrase "that individual's life" includes his or her private life, occupational life and social life with peers 
and relatives, as well as his or her life as a citizen of a community. 

A crucial capacity implied by this notion of mathematics is the ability to pose, formulate, solve and interpret 
problems using mathematics within a variety of situations and contexts. The contexts range from being purely 
mathematical to having no mathematical structure present or apparent at the outset - the problem poser 
or solver must successfully introduce the mathematical structure. It is also important to emphasise that the 
definition is not just concerned with knowing mathematics at some minimal level; it is also about doing and 
using mathematics in situations that range from the everyday to the unusual, from the simple to the complex. 



THEORETICAL BASIS FOR THE PISA MATHEMATICS FRAMEWORK 

The PISA definition of mathematical literacy is consistent with the broad and integrative theory about the 
structure and use of language as reflected in recent socio-cultural literacy studies. In James Gee's Preamble to 
a Literacy Program (1998), the term literacy refers to the human use of language. The ability to read, write, 
listen and speak a language is the most important tool through which human social activity is mediated. In 
fact, each human language and use of language has an intricate design tied in complex ways to a variety of 
functions. Being literate in a language implies that a person knows many of the design resources of the language 
and is able to use those resources for several different social functions. Analogously, considering mathematics 
as a language implies that students must learn the design features involved in mathematical discourse (the 
terms, facts, signs and symbols, procedures and skills to perform certain operations in specific mathematical 
sub-domains, and the structure of those ideas in each sub-domain), and they also must learn to use such ideas 
to solve non-routine problems in a variety of situations defined in terms of social functions. Note that the 
design features for mathematics include knowing the basic terms, procedures and concepts commonly taught 
in schools, and also involve knowing how these features are structured and used. Unfortunately, one can know 
a good deal about the design features of mathematics without knowing either their structure or how to use those 
features to solve problems. These scholarly notions involving the interplay of design features and functions that 
support the mathematics framework for PISA can be illustrated in the following example. 

MATHEMATICS EXAMPLE 1 : HEARTBEAT 

For health reasons people should limit their efforts, for instance during sports, in order not to exceed a certain 
heartbeat frequency. 

For years the relationship between a person's recommended maximum heart rate and the person's age was 
described by the following formula: 

Recommended maximum heart rate = 220 - age 

Recent research showed that this formula should be modified slightly. The new formula is as follows: 
Recommended maximum heart rate = 208 - (0.7 x age) 

The questions in this unit centre around the difference between the two formulae and how they affect the 
calculation of the maximum allowable heart rate. 
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Question 1: HEARTBEAT 

A newspgpef article stated; A result of using the new fofmulg instead of the ol4 one is thgt the 
recommended maximum numbef of hegrtbegts pef minute fof young people decreases slightly gnd 
fof old people it increases slightly." 

Ffom which gge onwgfds does the recommended maximum hegrt fgte increase gs g result of the 
introduction ofthe new fofmuUi’ Show youf wofk. 

Question 1 can be solved by following the general strategy used by mathematicians, which the mathematics 
framework will refer to as "mathematising". Mathematising can be characterised as having five aspects: 

■ First, the process of mathematlsation or mathematising starts with a problem situated in reality. 

As will be clear from the item, the issue situated in reality in this case is physical health and fitness: "An 
important rule when exercising is that one should be careful to not push oneself too far as excessive exertion 
may cause health problems." The question alerts us to this issue through the text linking health to heart rate and 
by referring to the "recommended maximum heart rate". 

■ Secondly, the problem solver tries to identify the relevant mathematics, and reorganises the problem according 
to the mathematical concepts identified. 

It seems clear that the student faces two word formulae that need to be understood, and he or she is requested 
to compare these two formulae and determine what they mean in mathematical terms. The formulae give a 
relation between the advised maximum heart beat rate and the age of a person. 

■ The third step involves gradually trimming away the reality. 

There are different ways of refocusing the problem to be a strictly mathematical problem, or of trimming away 
reality. One way would be to make the word formulae into more formal algebraic expressions like y = 220 - x 
or y = 208 - 0.7x. The student must remember that y expresses the maximum heart beat in beats per minute 
and X represents the age in years. Another strictly mathematical approach would be to draw the graphs directly 
from the word formulae. These graphs are straight lines as the formulae are of the first degree. The graphs have 
different slopes, so they intersect. 

These three steps lead one from a real-world problem to a mathematical problem. 



Maximum heartbeat 



Heartbeat 
— Old formula 
New formula 
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■ The fourth step is solving the mathematical problem. 

The mathematical problem at hand is to compare the two formulae, or graphs, and say something about the 
differences for people of a certain age. A nice way to start is to find out where the two formulae give equal 
results, or where the two graphs intersect. The student can find this by solving the equation: 220 - x = 208 - 
0.7x. This gives us x = 40 and the corresponding value for y is 1 80. So the two graphs intersect at the point (40, 
1 80). This point can also be found in the graph just above. As the slope of the first formula is -1 and the second 
is -0.7, the student knows that the second graph is less steep than the first. The student also knows that the graph 
line of y = 220 - x lies above the graph line of y = 208 - 0.7x for values of x smaller than 40 and lies below for 
values of x larger than 40. 

■ The fifth asks what the meaning of this strictly mathematical solution is in terms of the real world. 

The meaning is not too difficult if the student realises that x is the age of a person and y the maximum heart 
beat. If one is 40 years old both formulae give the same result: a maximum heartbeat of 180. The 'old' rule 
allows for higher heart rates for younger people: in the extreme, if the age is zero the maximum is 220 in the old 
formula and only 208 in the new formula. But for older people, in this case for those over 40, the more recent 
insights allow for higher maximum heartbeat; as an example: for an age of 100 years the student sees that the 
old formula gives him or her a maximum of 120 and the new one 138. Of course the student has to realise a 
number of other things: the formulae lack mathematical precision and are only quasi scientific. In reality, the 
formulae provide only a rule of thumb that should be used with caution. Moreover the outcomes for ages at the 
extreme should be regarded even more cautiously. 

What this example shows is that even with items that seem relatively simple in the sense that they can be used 
within the restrictions of a large international study and solved in a short time, the full cycle of mathematisation 
and problem solving can still be identified. 

It is these processes that characterise how, in a broad sense, mathematicians often do mathematics, how people 
use mathematics in a variety of current and potential occupations, and how informed and reflective citizens 
should use mathematics to fully and competently engage with the real world. In fact, learning to mathematise 
should be a primary educational goal for all students. 

Today and in the foreseeable future, every country needs mathematically literate citizens to deal with a very 
complex and rapidly changing society. Accessible information has been growing exponentially and citizens 
need to be able to decide how to deal with this information. Social debates increasingly involve quantitative 
information to support claims. Individuals often need to be mathematically literate to make judgements and 
assess the accuracy of conclusions and claims in surveys and studies. Being able to judge the soundness of the 
claims from such arguments is, and increasingly will be, a critical aspect of being a responsible citizen. The 
steps of the mathematisation process discussed in this framework are fundamental for one to use mathematics 
in such complex situations. Failure to use mathematical notions can result in confused personal decisions, an 
increased susceptibility to pseudo-sciences, and poorly informed decision-making in professional and public 
life. 

A mathematically literate citizen understands how quickly change is taking place and the consequent need to 
be open to lifelong learning. Adapting to these changes in a creative, flexible and practical way is a necessary 
condition for successful citizenship. The skills learned at school will probably not be sufficient to serve the 
needs of citizens for the majority of their adult life. 

The requirements for competent and reflective citizenship also affect the workforce. Workers are less frequently 
expected to carry out repetitive physical chores. Instead, they are engaged actively to monitor output from a 
variety of high-technology machines, deal with a flood of information and engage in team problem solving. In 
the future more occupations will require the ability to understand, communicate, use and explain concepts and 
procedures based on mathematical thinking. The steps of the mathematisation process are the building blocks 
of this kind of mathematical thinking. 

Mathematically literate citizens also develop an appreciation for mathematics as a dynamic, changing and 
relevant discipline that may often serve their needs. 
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The operational problem faced by PISA is how to assess whether 15-year-old students are mathematically 
literate in terms of their ability to mathematise. Unfortunately, in a timed assessment this is difficult because 
most complex real situations demand a considerable amount of time for one to collaborate and find appropriate 
resources while proceeding from reality to mathematics and back. 

To illustrate mathematisation in an extended problem-solving-exercise, consider the following HOLIDAY 
example, which was an item in the PISA 2003 problem-solving survey. The problem poses two questions to 
the students. It deals with the planning of a route and places to stay overnight on a holiday trip. Students were 
presented with a simplified map and a chart (multiple representations) showing the distances between the 
towns illustrated on the map. 

MATHEMATICS EXAMPLE 2: HOLIDAY 

This problem is about planning the best route for a holiday. 

Figures A and B show a map of the area and the distances between towns. 

Figure A. Map of roads between towns 
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Question 1 : HOLIDAY 

Cglculgte the shortest distance by togcj between Nuben gnd Kg4o. 
Distance: kilometres. 
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Question 2: HOLIDAY 

Zoe lives in Anggz. She wgnts to visit l<g4o gnd Upgt. She cgn only tfgvel up to 300 kilometres in gny 
one 49y, but cgn bfegk bef joufney by cgmping ovefnigbt anywhere between towns. 

Zoe will stgy fof two nights in egch town, so thgt she cgn spen4 one whole 4^y sightseeing in egch 
town. 

show Zoe's itineF^fy by completing the following tgble to in4ic^te where she stgys e^ch night. 



Day 


OveEnight Stay 


1 


Camp-site between Angaz an4 Ka4o 


2 




3 




4. 




5 




6 




7 


Angaz 



While there is no obvious link to a curricular discipline, there is a clear relation to discrete mathematics. There 
is also not a pre-described strategy to solve this problem. Often, students know exactly which strategy to use 
when posed a problem-solving exercise. But in real-world problem solving there is no well-known strategy 
available. 

This example also clearly presents the five aspects of mathematising. The problem is situated in reality and can 
be organised according to mathematical concepts (distance tables or matrices) and maps (as models of reality.) 
Additionally, the student is required to trim away redundant information and focus on relevant information, 
especially on the mathematical aspects of that information. Finally, the student needs to solve the problem in 
mathematical terms then reflect on the solution in terms of the real situation. 

Although there is relatively little reading required to solve the problem, it is still rather complex because students 
must read and interpret information from the map and the distance chart. Some of the distances that they have 
to find in the chart require them to read distances starting from the bottom of the chart and some starting from 
the left. For example, in determining the distance from Nuben to Piras, one needs to transform the search to that 
of finding the distance from Piras to Nuben (OECD, 2004). 

The second question sets a number of constraints that needed to be complied with simultaneously: a maximum 
of 300 kilometres travelled in a given day, starting and finishing in Zoe's hometown of Angaz, visiting Kado and 
Lapat, and staying two nights in each of these cities so that she can achieve her vacation goals. 

It should be noted than in the PISA problem-solving survey from which this item was taken, students were allowed 
considerably more time to find answers than normally allowed for the typically shorter mathematics items. 

Ideally, to judge whether 15-year-old students can use their accumulated mathematical knowledge to solve 
mathematical problems they encounter in their world, one would collect information about the students' abilities 
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to mathematise such complex situations. Since this is not a feasible option though, PISA instead prepares items 
to assess different parts of this process. The following section describes how a set of test items are created in a 
balanced manner so that all five aspects of mathematising are covered in the selection. The aim is to use the 
responses to those items to locate students on a scale of proficiency in the PISA assessment of mathematics 



ORGANISATION OF THE DOMAIN 

The PISA mathematics framework describes and provides the rationale for an assessment that examines 
how 15-year-old students handle mathematics in a well-founded manner when confronted with real-world 
problems, or more generally an assessment of how mathematically literate 1 5-year-old students are. To describe 
the domain more clearly, three components must be distinguished: 

■ the situations or contexts in which the problems are located; 

■ the mathematical content used to solve the problems, organised by certain overarching ideas; 

■ the mathematical competencies that must be activated to connect the real world, in which the problems are 
generated, with mathematics, and thus to solve the problems. 

These components are represented in Figure 2.1 . An explanation of each is provided afterwards. 

Figure 2.1 ■ The components of the mathematics domain 




A person's mathematical literacy is seen in the way he or she uses mathematical knowledge and skills to solve 
problems. Problems (and their solutions) may occur in a variety of situations or contexts within the experience 
of an individual. PISA problems draw from the real world in two ways. First, problems exist within some broad 
situations that are relevant to the student's life. The situations form part of the real world and are indicated by a 
big square in the upper left-hand corner of Figure 2.1 . Next, within that situation, problems have a more specific 
context. This is represented by the grey rectangle in the situations square. 

In the HEARTBEAT and HOLIDAY examples, both situations are the personal real world, and the contexts are 
sport/health aspects for the active citizen and planning a holiday. 

To solve real world problems such as these, a person must apply his or her knowledge of specific technical 
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mathematical content. Unlike a classroom situation, where the mathematical content under study is usually 
evident, typically the knowledge required when solving real-world problems is not immediately evident. The 
problem solver must engage with the phenomena through which the problem is presented, must identify what 
specific knowledge will likely be useful, and must activate that knowledge. 

PISA therefore identifies mathematical content by listing a small set of overarching ideas that represent broad 
categories of real-world phenomena through which opportunities to explore and use mathematics arise in 
our interactions with the world. This approach reflects the main threads in both the historical development 
of mathematics as a discipline, and the development of mathematical ideas in individuals. Those four broad 
categories are also elaborated to show the more specific mathematical objects, knowledge and skills typically 
used in relation to each, thereby showing the links between this approach to specifying mathematical content, 
and the topics covered in school mathematics curricula. 

The overarching ideas are represented by the big square in the upper right-hand corner of Figure 2.1. From 
the overarching ideas, the particular mathematical content used in solving a problem is extracted. This is 
represented by the smaller square within the overarching ideas square. 

The arrows going from the context and content to the problem show how the real world (including mathematics) 
makes up a problem. 

The HEARTBEAT problem involves mathematical relations and comparing two relations in order to make 
decisions. Thus, the problem belongs to the overarching idea change and relationships. The HOLIDAY problem 
requires some basic computation, and while the second question requires some analytic reasoning, the most 
appropriate overarching idea is quantity. 

Cognitive mathematical competencies are the mathematical processes that students apply as they attempt 
to solve problems. They encapsulate the different cognitive processes needed to solve various kinds of 
problems. The individual mathematical competencies as defined in the competency clusters reflect the way 
that mathematical processes are typically employed when solving problems that arise as students interact with 
their world. They will be described in detail in later sections. 

Thus the process component of this framework is represented first by the large square, representing the 
general mathematical competencies, and, inside that, a smaller square that represents the three clusters. The 
mathematical processes or competencies needed to solve a particular problem are related to the nature of the 
problem, and the competencies used will be reflected in the solution found. This interaction is represented 
by the arrow from the box containing the mathematical processes to the box containing the problem and its 
solution. 

The remaining arrow in the diagram goes from the box containing the mathematical processes to the box 
containing the problem format. The mathematical processes used to solve a problem are related to the form of 
the problem and its precise demands. 

It should be emphasised that the three components just described are of different natures. Indeed, the 
mathematical processes are the core of the mathematics assessment, and students will only be in a position 
to successfully solve problems when certain competencies are available to them. Assessing mathematical 
literacy includes assessing to what extent students possess mathematical knowledge and skills that they can 
productively apply in problem situations. 

In the following sections, these three components are described in more detail. 

Situations and context 

An important aspect of mathematical literacy is engagement with mathematics: using and doing mathematics 
in a variety of situations. It has been recognised that in dealing with issues that call for mathematical treatment, 
the choice of mathematical methods and representations is often dependent on the situations in which the 
problems are presented. 
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The situation is the part of the student's world in which the tasks are placed. It is located at a certain distance 
from the students. For PISA, the closest situation is the student's personal life. Next is school life, work life and 
leisure, followed by the local community and society as encountered in daily life. Scientific situations are 
furthest away. The four situation types defined and used for problems to be solved are: personal, educational/ 
occupational, public and scientific. 

The context of an item is its specific setting within a situation. It includes all the detailed elements used to 
formulate the problem. 

Consider the following example: 

MATHEMATICS EXAMPLE 3: SAVINGS ACCOUNT 

Question 1 : SAVINGS ACCOUNT 

1 000 ze4 is put into 3 savings account gt g b^nk. There ^re two choices: one cgn get gn ^nnugl rgte of 
4-% OR one cgn get gn immediate 10 ze4 bonus from the bgnk, gn4 3 3% gnnugl rgte. 

which option is better gfter one yegr/ After two yegrsi’ 

The situation of this item is finance and banking, a situation from the local community and society that PISA 
would classify as public. The context of this item concerns money (zeds) and interest rates for a bank account. 

This kind of problem is one that could be part of the actual experience or practice of the participant in some real- 
world setting. It provides an authentic context for the use of mathematics, since the application of mathematics in 
this context would be genuinely directed to solving the problem. This can be contrasted with problems frequently 
seen in school mathematics texts, where the main purpose is to practise the mathematics involved rather than to 
use mathematics to solve a real-world problem. This authenticity in the use of mathematics is an important aspect 
of the design and analysis of items for PISA, strongly related to the definition oi mathematical literacy. 

It should be noted that this use of the term authentic is not intended to indicate that mathematics items 
are genuine or real. PISA mathematics uses the term authentic to indicate that the use of mathematics is 
genuinely directed to solving the problem at hand, rather than the problem being merely a vehicle for the 
purpose of practising some mathematics. 

One should also note that some elements of the problem are made up, namely the money involved, the zeds. 
This fictitious element is introduced to ensure that students from certain countries are not given an unfair 
advantage. 

The situation and context of a problem can also be considered in terms of the distance between the problem 
and the mathematics involved. If a task refers only to mathematical objects, symbols or structures, and makes 
no reference to matters outside the mathematical world, the context of the task is considered to be intra- 
mathematical, and will be classified as belonging to the scientific situation type. A limited range of such tasks 
is included in PISA, where the close link between the problem and underlying mathematics is made explicit in 
the problem context. More typically, problems encountered in the day-to-day experience of the student are not 
stated in explicit mathematical terms. They refer to real-world objects. These task contexts are extra-mathematical 
and the student must translate these problem contexts into a mathematical form. Generally speaking, PISA puts 
an emphasis on tasks that might be encountered in some real-world situations and possess an authentic context 
for the use of mathematics that influences the solution and its interpretation. This, however, does not preclude 
the inclusion of tasks in which the context is hypothetical, as long as it has some real elements, is not too 
far removed from a real-world situation, and requires an authentic use of mathematics to solve the problem. 
Example 4 shows a problem with a hypothetical context that is extra-mathematical. 
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MATHEMATICS EXAMPLE 4: COINAGE SYSTEM 

Question 1 : COINAGE SYSTEM 

WouM it be possible to establish g coinage system bgse4 on only the denominations 3 gnd 5! More 
specifically, wbgt amounts could be fegcbed on tbgt bgsisi’ Would such g system be desif-gblei’ 

This problem does not necessarily derive its quality from its closeness to the real world; rather, it is 
mathematically interesting and calls on mathematical processes that are related to mathematical literacy. The 
use of mathematics to explain hypothetical scenarios and explore potential systems or situations is one of the 
most powerful features of this example, even if its actual scenarios or systems are unlikely to be carried out in 
reality. Such a problem would be classified as belonging to the scientific situation type. 

In summary, PISA places most value on tasks that could be encountered in a variety of real-world situations and 
have a context in which the use of mathematics to solve the problem would be authentic. Problems with extra- 
mathematical contexts that influence the solution and its interpretation are preferred as a vehicle for assessing 
mathematics since these problems are most like those encountered in day-to-day life. 

Mathematical content - the four overarching ideas 

Mathematical concepts, structures and ideas have been invented as tools to understand, organise, and analyse 
the phenomena of the natural, social and mental world. In schools, the mathematics curriculum has been 
logically organised around content strands (e.g., arithmetic, algebra, geometry) and their detailed topics that 
reflect historically well-established branches of mathematical thinking, and that facilitate the development of a 
structured teaching syllabus. However, in the real world the phenomena that lend themselves to mathematical 
treatment do not come so logically organised. Rarely do problems arise in ways and contexts that allow their 
understanding and solution to be achieved through the application of knowledge from a single content strand, 
and solving problems as they appear in the real world usually requires an expanded range of thought processes 
compared with those typically employed in the classroom. 

Since the goal of PISA is to assess students' capacity to solve real problems, our strategy has been to define 
the range of content that will be assessed using a phenomenological approach to describing mathematical 
concepts, structures or ideas. This means describing content in relation to the phenomena and the kinds of 
problems for which it was created. This approach ensures a focus in the assessment that is consistent with the 
domain definition, yet covers a range of content that includes what is typically found in other mathematics 
assessments and in national mathematics curricula. 

A phenomenological organisation for mathematical content is not new. Two well known publications On the 
Shoulders of Giants: New Approaches to Numeracy (Steen, 1 990) and Mathematics: The Science of Patterns 
(Devlin, 1 994) have described mathematics in this manner. Various terms and categorisations have been used in 
these and other similar publications. A choice is needed, and for PISA four have been chosen. The overarching 
ideas used for PISA assessment purposes reflect an understanding of the mathematics field that focuses on 
patterns. Patterns in quantity, patterns in space and shape and patterns in change and relationships form central 
and essential concepts for any description of mathematics, and they form the heart of any curriculum, whether at 
high school, college or university. But increasingly, dealing with uncertainty from a mathematical and scientific 
perspective is also seen to be essential. For this reason, elements of probability theory and statistics give rise to 
the fourth overarching idea, uncertainty. 

While this approach is somewhat different from the approach to content that would betaken from the perspective 
of mathematics instruction and the curricular strands typically taught in schools, nevertheless these overarching 
ideas encompass the full range of mathematical topics that students are expected to learn during their school 
mathematics studies. 
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The following list of overarching ideas, therefore, is used in PISA to meet the requirements of historical 
development, coverage of the domain and reflection of the major threads of school curriculum: 

■ space and shape 

■ change and relationships 

■ quantity 

■ uncertainty 

With these four overarching ideas, mathematical content can be organised into a sufficient number of areas that 
ensure a spread of items across the curriculum while still maintaining a sufficiently broad focus to facilitate the 
presentation of problems based in real situations. 

The basic conception of an overarching idea is an encompassing set of phenomena and concepts that make 
sense and can be encountered within and across a multitude of different situations. Each overarching idea can 
be perceived as a sort of general notion dealing with some generalised content dimension. This implies that 
neither can the overarching ideas, nor traditional mathematics content strands be sharply delineated in relation 
to one another. Rather, each idea represents a certain perspective or point of view and can be thought of as 
possessing a core, a centre of gravity, and somewhat blurred outskirts that allow for intersection with other 
overarching ideas. In principle, any overarching idea intersects any other overarching idea. The four overarching 
ideas are described in the following section. 

Space and shape 

Patterns are encountered everywhere: in spoken words, music, video, traffic, building constructions and art. 
Shapes can be regarded as patterns: of houses, office buildings, bridges, starfish, snowflakes, town plans, 
clover leaves, crystals or shadows. Geometric patterns can serve as relatively simple models of many kinds of 
phenomena, and their study is possible and desirable at all levels (Grunbaum, 1 985). 

It is important to have an understanding of the properties of objects and their relative positions. Students must be 
aware of how and why they see things and must learn to navigate through space and through constructions and 
shapes. This denotes an understanding of the relationship between shapes and images or visual representations, 
such as that between a real city and photographs and maps of the same city. It also includes an understanding 
of how three-dimensional objects can be represented in two dimensions, how shadows are formed and must 
be interpreted, and what perspective is and how it functions. 

Shape has strong ties to traditional geometry, but goes far beyond it in content, meaning and method. Interaction 
with real shapes involves understanding the visual world and its description and encoding and decoding visual 
information. It also means interpreting visual information. In order to grasp the concept of shape, students 
should be able to discover the ways in which objects are similar and different, analyse the different components 
of an object, and recognise shapes in different dimensions and representations. 

It is important to note that shapes can be more than just static entities. They can be transformed as entities, 
modified or sometimes visualised very elegantly using computer technology. Students should be able to see the 
patterns and regularities when shapes are changing. An example is shown in the following section in Figure 2.2. 

The study of shapes and constructions requires looking for similarities and differences when analysing the 
components of form and recognising shapes in different representations and dimensions. The study of shapes is 
closely connected to the concept of grasping space (Freudenthal, 1 973). 

Examples requiring this kind of thinking are abundant and include the following: identifying and relating a 
photograph of a city to a map of that city and indicating from which point a picture was taken; the ability 
to draw a map; understanding why a nearby building looks bigger than a building that is further away; or 
understanding how the rails of a railway track appear to meet at the horizon. All these examples are relevant 
for students within this overarching idea. 

As students live in a three-dimensional space, they should be familiar with views of objects from three orthogonal 
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aspects (for example, from the front, the side and above). They should be aware of the power and limitations 
of different representations of three-dimensional shapes, as indicated by the example provided in Figure 2.3. 
Students must not only understand the relative position of objects, but also how they can navigate through 
space and through constructions and shapes. An example is reading and interpreting a map and designing 
instructions on how to get from point A to point B using coordinates, common language or a picture. 

Conceptual understanding of shapes also includes the ability to take a three-dimensional object and make a 
two-dimensional net of it, and vice-versa, even if the three-dimensional object is presented in two dimensions. 
An example of this is given in Figure 2.4. 

Key aspects of space and shape are: 

■ recognising shapes and patterns in shapes; 

■ describing, encoding and decoding visual information; 

■ understanding dynamic changes to shapes; 

■ identifying similarities and differences; 

■ identifying relative positions; 

■ interpreting two-dimensional and three-dimensional representations and the relations between them; 

■ navigation through space. 

Space and shape examples 

Figure 2.2 offers a simple example of the need for flexibility in seeing shapes as they change. It is based on a 
cube that is being 'sectioned' (that is, plane cuts are made through the cube), and allows a variety of questions 
to be asked, such as: 

Whgt shapes cgn be pto4uce4 by one pUne cut through g cubef 

How mgny fgces, e4ges, or vertices will be pro4uce4 when 4 cube is sectione4 in this w^yf 

Figure 2.2 ■ A cube, with plane cuts in various places 




Three examples of the need for familiarity with representations of three-dimensional shapes follow. In the first 
example, the side and front views of an object constructed of cubes are given in Figure 2.3. The question is: 

How mgny cubes hgve been use4 to mgke this objectf 

Figure 2.3 ■ Side and front views of an object made from cubes 



Side view 




PISA 2009 ASSESSMENT FRAMEWORK - KEY COMPETENCIES IN READING, MATHEMATICS AND SCIENCE © OECD 2009 



I PISA 2009 MATHEMATICS FRAMEWORK 




It may come as a surprise to many - students and teachers alike - that the maximum number of cubes is 20 and 
the minimum is 6 (de Lange, 1 995). 



The next example shows a two-dimensional representation of a barn and an incomplete net of the barn. The 
problem is to complete the net of the barn. 



Figure 2.4- ■ Two-dimensional representation of a three-dimensional barn 

and its (incomplete) net 




A final example similar to the previous one is shown in Figure 2.5 (adapted from Hershkovitz et al., 1 996). 



Figure 2.5 ■ Cube with black bottom 




The lower half of the cube has been painted black. For each of the four nets, the bottom side is already black. 
Students could be asked to finish each net by shading the right squares. 

Change and relationships 

Every natural phenomenon is a manifestation of change, and the world around us displays a multitude of 
temporary and permanent relationships among phenomena. Examples include organisms changing as 
they grow; the cycle of seasons; the ebb and flow of tides; cycles of unemployment; weather changes; and 
stock exchange indices. Some of these processes of change involve and can be described or modelled by 
straightforward mathematical functions: linear, exponential, periodic or logistic, with any of these being either 
discrete or continuous. But many relationships fall into different categories, and data analysis is often essential 
in determining the kind of relationship that is present. Mathematical relationships often take the shape of 
equations or inequalities, but relations of a more general nature (e.g. equivalence, divisibility, inclusion, to 
mention but a few) may appear as well. 

In order to be sensitive to the patterns of change, Stewart (1 990) recommends: 

■ representing changes in a comprehensible form; 

■ understanding the fundamental types of change; 

■ recognising particular types of change when they occur; 



96 
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■ applying these techniques to the outside world; 

■ controlling a changing universe to the best advantage. 

Change and relationships can be represented in a variety of ways including numerical (for example, in a table), 
symbolic, graphical, algebraic and geometrical. Translation between these representations is of key importance, 
as is the recognition of an understanding of fundamental relationships and types of change. Students should 
be aware of the concepts of linear growth (additive process), exponential growth (multiplicative process) and 
periodic growth, as well as logistic growth (at least informally as a special case of exponential growth). 

Students should also see the relationships among these models - the key differences between linear and 
exponential processes, the fact that percentage growth is a form of exponential growth or how logistic growth 
occurs and why, either in continuous or discrete situations. 

Changes occur in a system of interrelated objects or phenomena when the elements influence each other. In the 
examples mentioned in the summary, all phenomena change over time. But there are many real-life examples 
of matters in which objects are interrelated in a multitude of ways. For example: 

If the length of the string of a guitar is halved, the new tone is an octave higher than the original tone. 
The tone is therefore dependent on the string length. 

When we deposit money into a bank account, we know that the account balance will depend on the 
size, frequency and number of deposits and withdrawals, and the interest rates. 

Relationships lead to dependency. Dependency concerns the fact that properties and changes of certain 
mathematical objects may depend on or influence properties and changes of other mathematical objects. 
Mathematical relationships often take the form of equations or inequalities, but relations of a more general 
nature may appear as well. 

Change and relationships involves functional thinking. Functional thinking - that is, thinking in terms of 
and about relationships - is one of the most fundamental disciplinary aims of the teaching of mathematics 
(MAA, 1923). For 15-year-old students, this includes having a notion of rate of change, gradients and steepness 
(although not necessarily in a formal way), and dependence of one variable on another. Students should be able 
to make judgements about how fast processes are taking place in a relative way as well. 

This overarching idea closely relates to aspects of other overarching ideas. The study of patterns in numbers can 
lead to intriguing relationships such as the study of Fibonacci numbers or the Golden Ratio. The Golden Ratio is 
a concept that plays a role in geometry as well, thus relating closely to the overarching idea of space and shape. 
Many more examples of change and relationships can be found in space and shape, such as with the growth 
of an area in relation to the growth of a perimeter or diameter. Euclidean geometry also lends itself to the study 
of relationships. A well-known example is the relationship between the three sides of a triangle. If the lengths 
of two sides are known, the third is not determined but the interval in which it lies is known; the interval's end 
points are the absolute value of the difference between the other two sides, and their sum, respectively. Several 
other similar relationships exist for the various elements of a triangle. 

Uncertainty lends itself to various problems that can be viewed from the perspective of change and relationships. 
For example, if two fair dice have been rolled and one of them shows four, what is the chance that the sum 
exceeds seven? The answer (50%) relies on the dependency of the probability at issue on the set of favourable 
outcomes. The required probability is the proportion of all such outcomes compared with all possible outcomes, 
which is a functional dependency. 
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Change and relationships examples 

MATHEMATICS EXAMPLE 5: SCHOOL EXCURSION 

A school cUss wgnts to rent 9 cogcb fof gn excufsion, gnd three companies gre contacted fof 
infofmgtion gbout prices. 

Company A charges gn initial rgte of 375 ze4 plus 0.5 ze4 per kilometre 4riven. Company B charges 
gn initial rgte of 250 ze4 plus 0.75 ze4 per kilometre 4riven. Company C charges g flgt rgte of 350 ze4 
up to 200 kilometres, plus 102 ze4 per kilometre beyon4 200 km. 

which company shoul4 the cUss choose, ifthe excursion involves g totgl travel 4istgnce of 
somewhere between 400 gn4 600 kmi” 

Leaving aside the fictitious elements of the context, a problem like this could conceivably occur in the real 
world. Its solution requires the formulation and activation of several functional relationships, and equations 
and inequations. It can be dealt with by graphical as well as algebraic means, or combinations of both. The 
fact that the total travel distance in the excursion is not exactly known also introduces links to the uncertainty 
overarching idea, discussed in a later section. 

A graphical representation of the problem is presented in Figure 2.6. 

Figure 2.6 ■ Excursion charges for three bus companies 




The following is another example of change and relationships. 
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MATHEMATICS EXAMPLE 6: CELL GROWTH 



Poctofs gre monitofing the growth of cells. They gro pgrticuUfly interested in the d^y thgt the cell 
count will regch 60 000 because then they hgve to stgrt gn expefiment. The tgble of results is; 



Time 

Cdays) 


4. 


6 


8 


10 


12 


U 


16 


18 


20 


22 


24 


Cells 


597 


893 


1 339 


1 995 


2 976 


4 4-34. 


6 606 


9 878 


14 719 


21 956 


32 763 



Question 1 : CELL GROWTH 

when will the number of cells regch 60 OOOi” 

MATHEMATICS EXAMPLE 7: PREY-PREDATOR 

The following grgph shows the growth oftwo living organisms: the Pgrgmecium gnd Sgcchgromyces.; 




Days 

One ofthe two gnimgis fpred^tor) egts the other one fprey). Looking gt the grgph, cgn you judge 
which one is the prey gnd which one the pred^tori” 

One property of prey-predator phenomena is expressed 9 S; The rgte of growth of predators is 
proportional to the number of gvgiigble prey. Poes this property hold for the ^bove grgphsi’ 

Quantity 

Important aspects of quantity include an understanding of relative size, recognition of numerical patterns, and 
use of numbers to represent quantities and quantifiable attributes of real-world objects (counts and measures). 
Quantity additionally deals with the processing and understanding of numbers that are represented to us in 
various ways. 

An importantaspectof dealing with quantity is quantitative reasoning. Essential components of quantitative reasoning 
are number sense, representing numbers in various ways, understanding the meaning of operations, having a feel for 
the magnitude of numbers, mathematically elegant computations, mental arithmetic and estimating. 

Some of the most important and frequent uses of numbers in every-day life are seen when magnitudes are 
measured: length, area, volume, height, speed, mass, air pressure, money value are all quantified using measures. 
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Understanding the meaning of operations includes the ability to perform operations involving comparisons, 
ratios and percentages. Number sense addresses issues of relative size, different representations of numbers, 
equivalent form of numbers and using understanding of these things to describe attributes of the world. 

Quantity also includes having a feeling for quantities and estimation. In order to be able to test numerical results 
for reasonableness, one needs a broad knowledge of quantities (measures) in the real world. Is the average 
speed of a car 5, 50 or 500 km/h? Is the population of the world 6 million, 600 million, 6 billion, or 60 billion? 
How tall is a tower? How wide is a river? The ability to make quick order-of-magnitude approximations is of 
particular importance, especially when viewed in light of the increasing use of electronic calculating tools. 
One needs to be able to see that 33 x 61 3 is something around 20 000. To achieve this skill one does not need 
extensive training in mental execution of traditional written algorithms, but a flexible and smart application of 
place value understanding and single-digit arithmetic (Fey, 1 990). 

By using number sense in an appropriate way, students can solve problems requiring direct, inverse and joint 
proportional reasoning. They are able to estimate rates of change and provide a rationale for the selection of 
data and level of precision required by operations and models they use. They can also examine alternative 
algorithms, showing why they work or in what cases they fail. They can develop models involving operations, 
and relationships between operations, for problems involving real-world data and numerical relations requiring 
operations and comparisons (Dossey, 1997). 

In the overarching idea of quantity, there is a place for 'elegant' quantitative reasoning like that used by Gauss, 
as discussed in the following example. Creativity coupled with conceptual understanding should be valued at 
the level of schooling that includes 15-year-old students. 

Quantity examples 

MATHEMATICS EXAMPLE 8: GAUSS 

Kqtl Ftiedtich Gguss' C1777-1855) tegchet 3sked the cigss to gdd together gll the numbers from 1 
to 100. Presumably the tegcher's gim wgs to keep the students occupied for g time. But Gguss wgs gn 
excellent qugntitgtive regsoner gnd spotted g short-cut to the solution. His regsoning went like this; 

You write down the sum twice, once in gscending order, then in descending order, like this; 

1 + 2 + 3 + + 98 + 99 + 100 

100 + 99 + 98+ + 3 + 2+1 

Now you gdd the two sums, column by column, to give; 

101 +101 + + 101+101 

As there gre exgctly 100 copies ofthe number 101 in this sum its vglue is; 100 x 101 = 10 100. 

Since this product is twice the gnswer to the origingl sum, if you hgive it, you obtgin the 
gnswer; 5 050. 

Triangular numbers 

This example of quantitative thinking involving patterns of numbers can be taken a little further to demonstrate 
a link with a geometric representation of that pattern. The following formula gives the general situation for 
Gauss's problem: 

1 -I- 2 + 3 ++ n= n(n + 1)/2 

This formula also captures a geometric pattern that is well known: numbers of the form n(n+1)/2 are called 
triangular numbers since they are precisely the numbers that are obtained when balls are arranged in an 
equilateral triangle. 
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Figui^e 2.7 ■ The first five triangular numbers 



• • •x • • • • 

^ •• ••• 
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The first five triangular numbers 1, 3, 6, 10 and 1 5 are shown in Figure 2.7. 

Proportional reasoning 

It will be interesting to see how students in different countries solve problems that lend themselves to the use 
of a variety of strategies. Differences can be expected especially in the area of proportional reasoning. In some 
countries, one strategy per item is likely to be used, while in other countries more strategies will be used. Also, 
similarities in reasoning will appear in solving problems that do not look very similar. This is in line with recent 
research results from TIMSS data (Mitchell, J. et at., 2000). The following three items illustrate this point about 
different strategies and the relationships among them: 

1 . Tonight you're giving a party. You want to buy 1 00 cans of soft drink. How many six-can packs are you 
going to buy? 

2. A hang-glider with glide-ratio 1 to 22 starts from a sheer cliff at 120 metres. The pilot is aiming at a spot 
at a distance of 1 400 metres. Will she reach that spot (under conditions of no wind)? 

3. A school wants to rent mini-vans (with seats for eight passengers) for going to a school camp and 98 
students need transportation. How many vans does the school need? 

The first problem could be seen as a division problem (1 00-^6=) thatthen leaves the student with an interpretation 
problem back to the context (what is the meaning of the remainder?). The second problem can be solved by 
proportional reasoning (for every metre height I can fly a distance of 22 metres, so starting from 120 metres...). 
The third problem will be solved by many as a division problem. All three problems, however, can be solved 
using the ratio table method: 



Bottles : 



Flying : 



Buses : 



1 


10 


5 


15 


2 


17 


6 


60 


30 


90 


12 


102 



1 1 


100 


20 


120 


22 


2200 


440 


2640 


1 1 


10 


2 


13 


1 ® 


80 


16 


104 



Identifying this similarity is a skill that belongs to mathematics: mathematically literate students do not need 
to look for the one available and appropriate tool or algorithm, but can choose from a wide array of strategies. 
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MATHEMATICS EXAMPLE 9: PERCENTS 

Carl went to a store to buy a jacket with a normal price of 50 zed that was on sale for 20% off. In Zedland 
there is a 5% sales tax. The clerk first added the 5% tax to the price of the jacket and then took 20% off. Carl 
protested: he wanted the clerk to deduct the 20% discount first and then calculate the 5% tax. 

Question 1 : PERCENTS 

Poes it mgke gny difference!’ 

Problems involving this kind of quantitative thinking, and the need to carry out the resulting mental calculations, 
are encountered frequently when shopping. The ability to effectively handle such problems is fundamental to 
mathematics. 

Uncertainty 

Science and technology rarely deal with certainty. Indeed, scientific knowledge is seldom, if ever, absolute - 
and is even sometimes wrong - so some uncertainty always remains in even the most scientific predictions. 
Uncertainty is also present in daily life: uncertain election results, collapsing bridges, stock market crashes, 
unreliable weather forecasts, poor predictions for population growth, or economic models that don't align. 

As an overarching idea, uncertainty suggests two related topics: data and chance. These phenomena are 
respectively the subject of mathematical study in statistics and probability. Relatively recent recommendations 
concerning school curricula are unanimous in suggesting that statistics and probability should occupy a much 
more prominent place than in the past (Committee of Inquiry into the Teaching of Mathematics in Schools, 
1982; LOGSE, 1990; MSEB, 1990; NCTM, 1989; NCTM, 2000). Specific mathematical concepts and activities 
that are important in this area include collecting data, data analysis and display/visualisation, probability and 
inference. 

The recommendations on the role of data, statistics and probability in school curricula emphasise data analysis. 
As a result, it is easy to view statistics in particular as a collection of specific skills. David S. Moore has pointed 
out what the overarching idea uncertainty really entails. The PISA definition will follow his ideas as presented in 
On the Shoulders of Giants (Steen, 1990) and F. James Rutherford's ideas as presented in Why Numbers Count 
(Steen, 1997). 

Statistics brings something to mathematics education that is unique and important: reasoning from uncertain 
empirical data. This kind of statistical thinking should be part of the mental equipment of every intelligent 
citizen. The core elements are the: 

■ omnipresence of variation in processes; 

■ need for data about processes; 

■ design of data production with variation in mind; 

■ quantification of variation; 

■ explanation of variation. 

Data are not merely numbers, but numbers in a context. Data are obtained by measurement and represented 
by a number. Thinking about measurement leads to a mature grasp of why some numbers are informative and 
others are irrelevant or nonsensical. 

The design of sample surveys is a core topic in statistics. Data analysis emphasises understanding the specific 
data at hand, assuming they represent a larger population. The concept of simple random samples is essential 
for 15-year-old students to understand the issues related to uncertainty. 

Phenomena have uncertain individual outcomes and the pattern of repeated outcomes is often random. The 
concept of probability in the present PISA study will generally be based to situations regarding chance devices 
like coins, number cubes and spinners, or not too complex real-world situations that can be analysed intuitively, 
or can feasibly be modelled with these devices. 
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Uncertainty also appears from sources like natural variation in students' heights, reading scores, incomes of 
a group of people, etc. A very important step, even for 15-year-old students, is to see the study of data and 
chance as a coherent whole. One such principle is the progression of ideas from simple data analysis to data 
production to probability to inference. 

The important specific mathematical concepts and activities in this area are: 

■ producing data 

■ data analysis and data display/visualisation 

■ probability 

■ inference 

Uncertainty examples 

The following examples illustrate the uncertainty overarching idea. 

MATHEMATICS EXAMPLE 1 0: AVERAGE AGE 

if 4-0% oftbe population of g country gre gt legst 60 yegrs olcj, is it then possible for the gvergge gge 
to be 30 f 

MATHEMATICS EXAMPLE 1 1 : GROWING INCOMES? 

Hgs the income of people in ZedUnd gone up or 4own in recent 4ecg4esi’ The me4i3n money income 
per bousebol4 fell: in 1970 it wgs 34 2 00 ze4, in 1980 it wgs 30 500 ze4 9n4 in 1990 31 200 ze4- But 
the income per person incregse4: in 1970 13 500 ze4, in 1980 13 850, gn4 in 1990 15 777 ze4- 

A bousebol4 consists of gll people living together gt the sgme g44ress. Explain bow it is possible for 
the bouseboM income to go 4own gt be sgme time the per-person income bgs risen in Ze4lgn4. 

MATHEMATICS EXAMPLE 1 2: RISING CRIMES 



The following graph was taken from the weekiv Zedland News Masazine: 




It shows the number of reported crimes per 1 00 000 inhabitants, starting with five-year intervals, then changing 
to one-year intervals. 
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Question 1 : RISING CRIMES 

How mgny reported cn'mes pef 100 000 were there in 1960i’ 

Manufacturers of alarm systems used the same data to produce the following graph: 




Crime triples 
STOP 

the increase 
BUY ALARM SYSTEMS ■ 



Question 2: RISING CRIMES 

How did the designers come up with this grgph gnd whyi” 

The police were not too hgppy with the grgph from the gigrm systems manufacturers because the 
police wgnt to show how successful crime fighting hgs been. 

Pesign 3 grgph to be used by the police to demonstrate thgt crime h^s decreased recently. 
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Mathematical processes 

Mathematisation 

PISA examines the ability of students to analyse, reason and communicate mathematical ideas effectively 
as they pose, formulate, solve and interpret mathematical problems in a variety of situations. Such problem 
solving requires students to use the mathematical processes, knowledge and skills they have acquired through 
schooling and life experiences. In PISA, the fundamental process that students use to solve real-life problems is 
referred to as mathematisation. 



Figure 2.8 ■ The mathematisation cycle 



Real world Mathematical world 




The discussion above of the theoretical basis for the PISA mathematics framework outlines a five-step description 
of mathematisation. These steps are shown in Figure 2.8 and listed below: 

1 .Starting with a problem situated in reality. 

2. Organising it according to mathematical concepts and identifying the relevant mathematics involved. 

3. Gradually trimming away the reality through processes such as making assumptions, generalising and 
formalising. These processes promote the mathematical features of the situation and transform the real-world 
problem into a mathematical problem that faithfully represents the situation. 

4. Solving the mathematical problem. 

5. Making sense of the mathematical solution in terms of the real situation, including identifying the limitations 
of the solution. 

Mathematisation first involves translating the problem from reality into mathematics. This process includes 
activities such as: 

■ Identifying the relevant mathematics with respect to a problem situated in reality. 

■ Representing the problem in a different way, including organising it according to mathematical concepts and 
making appropriate assumptions. 

■ Understanding the relationships between the language of the problem and the symbolic and formal language 
needed to understand it mathematically. 

■ Finding regularities, relations and patterns. 

■ Recognising aspects that are isomorphic with known problems. 

■ Translating the problem into mathematics i.e. to a mathematical model (de Lange, 1987). 

As soon as a student has translated the problem into a mathematical form, the whole process can continue 
within mathematics. Students pose questions like: "Is there...?", "If so, how many?", "How do I find...?", 
using known mathematical skills and concepts. They attempt to work on their model of the problem 
situation, adjust it, establish regularities, identify connections and create a good mathematical argument. 
This part of the mathematisation process is generally called the deductive part of the modelling cycle 
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(Blum, 1 996; Schupp, 1 988). However, processes other than strictly deductive ones may play a part in this 
stage as well. This part of the mathematisation process includes: 

■ Using and switching between different representations. 

■ Using symbolic, formal and technical language and operations. 

■ Refining and adjusting mathematical models, combining and integrating models. 

■ Argumentation. 

■ Generalisation. 

The last step or steps in solving a problem involve reflecting on the whole mathematisation process and the 
results. Here students must interpret the results with a critical attitude and validate the whole process. Such 
reflection takes place at all stages of the process, but it is especially important at the concluding stage. Aspects 
of this reflecting and validating process are: 

■ Understanding the extent and limits of mathematical concepts. 

■ Reflecting on mathematical arguments and explaining and justifying results. 

■ Communicating the process and solution. 

■ Critiquing the model and its limits. 

This stage is indicated in two places in Figure 2.8 by the label 5 , where the mathematisation process passes 
from the mathematical solution to the real solution and is related back to the original real-world problem. 

The cognitive mathematical competencies 

The previous section focused on the major concepts and processes involved in mathematisation. An individual 
who is to engage successfully in mathematisation within a variety of situations, extra- and intra-mathematical 
contexts, and overarching ideas, needs to possess a number of mathematical competencies which, taken 
together, can be seen as constituting comprehensive mathematical competence. Each of these competencies 
can be possessed at different levels of mastery. Different parts of mathematisation draw differently upon these 
competencies, with regard to both the particular ones involved and the level of mastery required. PISA has 
decided to make use of eight characteristic cognitive mathematical competencies that rely, in their present 
form, on the work of Niss (1999) and his Danish colleagues. Similar formulations may be found in the work 
of many others (as indicated in Neubrand et a/., 2001 ). Some of the terms used, however, are used differently 
among different authors. 

■ Thinking and reasoning: this involves posing questions characteristic of mathematics ("Is there...?", "If so, 
how many?", "How do I find...?"); knowing the kinds of answers mathematics can offer to such questions; 
distinguishing between different kinds of statements (definitions, theorems, conjectures, hypotheses, examples, 
conditioned assertions); and understanding and hand ling the extent and limits of given mathematical concepts. 

m Argumentation: this involves knowing what mathematical proofs are and how they differ from other kinds 
of mathematical reasoning; following and assessing chains of mathematical arguments of different types; 
possessing a feel for heuristics ("What can or cannot happen, and why?"); and creating and expressing 
mathematical arguments. 

■ Communication: this involves expressing oneself, in a variety of ways, on matters with a mathematical 
content, in oral as well as in written form, and understanding others' written or oral statements about such 
matters. 

■ Modelling: this involves structuring the field or situation to be modelled; translating reality into mathematical 
structures; interpreting mathematical models in terms of reality; working with a mathematical model; 
validating the model; reflecting, analysing and offering a critique of a model and its results; communicating 
about the model and its results (including the limitations of such results); and monitoring and controlling the 
modelling process. 
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■ Problem posing and solving: this involves posing, formulating and defining different kinds of mathematical 
problems (for example "pure", "applied", "open ended" and "closed"), and solving different kinds of 
mathematical problems in a variety of ways. 

■ Representation: this involves decoding and encoding, translating, interpreting and distinguishing between 
different forms of representation of mathematical objects and situations; the interrelationships between the 
various representations; and choosing and switching between different forms of representation, according to 
situation and purpose. 

■ Using symbolic, formal and technical language and operations: this involves decoding and interpreting 
symbolic and formal language, and understanding its relationship to natural language; translating from 
natural language to symbolic/formal language; handling statements and expressions containing symbols and 
formulae; and using variables, solving equations and undertaking calculations. 

■ Use of aids and tools: this involves knowing about and being able to make use of various aids and tools 
(including information technology tools) that may assist mathematical activity and knowing about the 
limitations of such aids and tools. 

PISA does not test these cognitive mathematical competencies individually. There is considerable overlap among 
them, and when using mathematics, it is usually necessary to draw simultaneously on many of them, so that 
any effort to assess individual ones is likely to result in artificial tasks and unnecessary compartmentalisation. 
The particular mathematical processes, knowledge and skills students are able to display vary considerably 
among individuals. This is partially because all learning occurs through experiences, "with individual 
knowledge construction occurring through the processes of interaction, negotiation, and collaboration (de 
Corte, Greer and Verschaffel, 1996). PISA assumes that much of students' mathematics is learned in schools 
and acknowledges that an understanding of the domain is acquired gradually. More formal and abstract ways 
of representing and reasoning emerge over time as a consequence of engagement in activities designed to help 
informal ideas evolve. Mathematical understanding is also acquired through experience involving interactions 
in a variety of social situations or contexts. 

Some structure is needed in order to productively describe and report students' capabilities, as well as their 
strengths and weaknesses from an international perspective. One way of providing this in a comprehensible 
and manageable way is to describe clusters of cognitive mathematical competencies, based on the kinds of 
cognitive demands needed to solve different mathematical problems. 

Competency clusters 

PISA has chosen to describe the cognitive activities that these cognitive mathematical competencies encompass 
according to three clusters: the reproduction cluster, the connections cluster and the reflection cluster. In the 
following sections the three clusters are described and the ways in which the individual competencies are 
played out in each cluster are discussed. 

The reproduction cluster 

The mathematical competencies in this cluster involve the reproduction of practised knowledge. They include 
those mathematical processes, knowledge and skills most commonly targeted on standardised assessments 
and classroom tests. These are knowledge of facts and of common problem representations, recognition of 
equivalents, recollection of familiar mathematical objects and properties, performance of routine procedures, 
application of standard algorithms and technical skills, manipulation of expressions containing symbols and 
formulae in standard form, and carrying out computations. 

M Thinking and reasoning: this involves posing the most basic forms of questions ( "How many...?", "How much 
is...?") and understanding the corresponding kinds of answers ("so many...", "this much..."); distinguishing 
between definitions and assertions; understanding and handling mathematical concepts in the sorts of 
contexts in which they were first introduced or have subsequently been practised. 

■ Argumentation: this involves following and justifying standard quantitative processes, including computational 
processes, statements and results. 
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m Communication: this involves understanding and expressing oneself orally and in writing about simple 
mathematical matters, such as reproducing the names and the basic properties of familiar objects, citing 
computations and their results, usually not in more than one way. 

m Modelling: this involves recognising, recollecting, activating and exploiting well-structured, familiar 
models; interpreting back and forth between such models (and their results) and reality; and elementary 
communication about model results. 

■ Problem posing and solving: this involves posing and formulating problems by recognising and reproducing 
practised standard pure and applied problems in closed form; and solving such problems by invoking and 
using standard approaches and procedures, typically in one way only. 

■ Representation: this involves decoding, encoding and interpreting familiar, practised standard representations 
of well known mathematical objects. Switching between representations is involved only when the switching 
itself is an established part of the representations implied. 

■ Using symbolic, formal and technical language and operations: this involves decoding and interpreting routine 
basic symbolic and formal language practised in well known contexts and situations; and handling simple 
statements and expressions containing symbols and formulae, including using variables, solving equations 
and undertaking calculations by routine procedures. 

■ Use of aids and tools: this involves knowing about and being able to use familiar aids and tools in contexts, 
situations and ways close to those in which their use was introduced and practised. 

Assessment items measuring the reproduction cluster can be described with the following key descriptors: 

reproducing practised material and performing routine operations. 

The following are examples of reproduction cluster items that could be used in the assessment: 



MATHEMATICS EXAMPLE 1 3 

Solve the ecjugtion 7x-5 = 13x+15 

MATHEMATICS EXAMPLE 1 4 

Whgt is the gvergqe of 7, 12, 8, 14, 15, 91 



MATHEMATICS EXAMPLE 1 5 

1000 ze4 is put in g savings account gt g bgnk, with gn interest r^te of 4%. How mgny ze4 will there 
be in the account gfter one yegrf 

REACTION TIME, is an example of a PISA item used in the 2003 Field Trial. EXPORTS was used in the 2003 
main survey. 



MATHEMATICS EXAMPLE 1 6: REACTION TIME 




In a sprinting event, the reaction time is the time interval between the starter's 
gun firing and the athlete leaving the starting block. The final time includes both 
this reaction time, and the running time. 
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The following table gives the reaction time and the final time of 8 runners in a 100 metre sprint race. 



Lane 


Reaction time (sec) 


Final time (sec) 


1 


0.147 


10.09 


2 


0.136 


9.99 


3 


0.197 


9.87 


4 


0.180 


Did not finish the race 


5 


0.210 


10.17 


6 


0.216 


10.04 


7 


0.174 


10.08 


8 


0.193 


10.13 



Question 1 : REACTION TIME 

Identify the Gold, Silver gnd Bronze medallists from this rgce. Fill in the tgble below with the 
medallists' Igne number, reaction time gnd fingl time. 



Medal 


Lane 


Reaction time fsec) 


Final time Csec) 


GOLD 








SILVER 








BRONZE 









MATHEMATICS EXAMPLE 1 7: EXPORTS 

The graphics show information about exports from Zedland, a country that uses zeds as its currency. 
45 
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Question 1 : EXPORTS 

Wbgt wgs the vglue of fruit juice exported from Zedignd in 2000! 

A. 18 million zeds. 

B. 2.3 million zeds. 

C. 2.4 million zeds. 

D. 3.4 million zeds. 

E. 3.8 million zeds. 
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In order to clarify the boundary for items from the reproduction cluster, the SAVINGS ACCOUNT problem 
described in Example 3 provides an example that does not belong to the reproduction cluster. This problem 
takes most students beyond the simple application of a routine procedure and requires the application of a 
chain of reasoning and sequence of computational steps that are not characteristic of mathematical processes, 
knowledge and skills in the reproduction cluster. 

The connections cluster 

The connections cluster builds on the reproduction cluster by applying problem solving to situations that are 
not routine but still involve familiar or quasi-familiar settings. These mathematical processes knowledge and 
skills include the following: 

■ Thinking and reasoning: this involves posing questions ("How do I find...?", "Which mathematics is 
involved...?") and understanding the corresponding kinds of answers (provided by means of tables, graphs, 
algebra, figures, etc.); distinguishing between definitions and assertions and between different kinds of 
assertions; and understanding and handling mathematical concepts in contexts that are slightly different from 
those in which they were first introduced or have subsequently been practised. 

m Argumentation: this involves simple mathematical reasoning without distinguishing between proofs and 
broader forms of argument and reasoning; following and assessing chains of mathematical arguments of 
different types; and possessing a feel for heuristics (e.g. What can or cannot happen, or be the case, and 
why?", "What do I know and what do I want to obtain?"). 

■ Communication: this involves understanding and expressing oneself orally and in writing about mathematical 
matters ranging from reproducing the names and basic properties of familiar objects and explaining 
computations and their results (usually in more than oneway), to explaining matters that include relationships. 
It also involves understanding others' written or oral statements about such matters. 

■ Modelling: this involves structuring the field or situation to be modelled; translating reality into mathematical 
structures in contexts that are not too complex but nevertheless different from what students are usually 
familiar with. It also involves interpreting back and forth between models (and their results) and reality, 
including aspects of communication about model results. 

m Problem posing and solving: this involves posing and formulating problems beyond the reproduction of 
practised standard pure and applied problems in closed form and solving such problems by invoking and 
using standard approaches and procedures, as well as more independent problem-solving processes in which 
connections are made between different mathematical areas and modes of representation and communication 
(schemata, tables, graphs, words, pictures). 

■ Representation: this involves decoding, encoding and interpreting familiar and less familiar representations 
of mathematical objects; choosing and switching between different forms of representation of mathematical 
objects and situations; and translating and distinguishing between different forms of representation. 

■ Using symbolic, formal and technical language and operations: This involves decoding and interpreting basic 
symbolic and formal language in less well-known contexts and situations, and handling statements and 
expressions containing symbols and formulae, including using variables, solving equations and undertaking 
calculations using familiar procedures. 

■ Use of aids and tools: this involves knowing about and using familiar aids and tools in contexts, situations and 
ways that are different from those in which their use was introduced and practised. 

Items associated with this cluster usually require some evidence of the integration and connection of material 
from the various overarching ideas, or from different mathematical curriculum strands, or the linking of different 
representations of a problem. 

Assessment items measuring the connections cluster might be described with the following key descriptors: 
integrating, connecting and modest extension of practised material. 

Examples of connections cluster items 

One example of a connections cluster item was given in the SAVINGS ACCOUNT problem described in 
Example 3. Other examples of connections cluster items follow. 
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Mary lives two kilometres from school, Martin five. 
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Question 1 : DISTANCE 

How fgr <\o Mgry grid Mgrtin live from egch otherf 

When this problem was originally presented to teachers, many of them rejected it on the grounds that it was too 
easy - one could easily see that the answer was three. Another group of teachers argued that it was not a good 
item because there was no answer - meaning there was not one single numerical answer. A third reaction was 
that it was not a good item because there were many possible answers since without further information, the 
most that could be concluded was that Mary and Martin live somewhere between three and seven kilometres 
from one another, and that was not desirable for an item. A small group thought it was an excellent item 
because one must understand the question, it is real problem solving because there is no strategy known to the 
student, and it is beautiful mathematics, although you have no clue how students will solve the problem. It is 
this last interpretation that associates the problem with the mathematical processes, knowledge and skills in the 
connections cluster. 



MATHEMATICS EXAMPLE 1 9: THE OFFICE RENTING 



The following two advertisements appeared in a daily newspaper in a country where the units of currency 
are zeds. 



BUILDING A 


BUILDING B 


Office space available 


Office space available 


58-95 square metres 
475 zeds per month 


35-260 square metres 
90 zeds per square metre 


1 00-1 20 square metres 
800 zeds per month 


per year 



Question 1 : THE QFFICE RENTING 

if 9 company is interested in renting 9n ofj^ce ofl10 sgugre metres in thgt country for 9 ye9r, 9t which 
office building, A or B, should the comp9ny rent the office in order to get the lower pricei’ Show your 
work. I© lEA/TIMSS] 

MATHEMATICS EXAMPLE 20: THE PIZZA 

A pizzeria serves two round pizzas of the same thickness in different sizes. The smaller one has a diameter of 30 
cm and costs 30 zeds. The larger one has a diameter of 40 cm and costs 40 zeds. [© PRIM, Stockholm Institute 
of Education] 

Question 1 : THE PIZZA 

which pizz9 is better V9lue for money i’ Show your re9Soning. 

In both of these problems, students are required to translate a real-life situation into mathematical language, to 
develop a mathematical model that enables them to make a suitable comparison, to check that the solution fits 
in with the initial question context and to communicate the result. These are all activities associated with the 
connections cluster. 
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The reflection cluster 

The mathematical processes, knowledge and skills in this cluster include an element of reflectiveness on the 
part of the student about the processes needed or used to solve a problem. They relate to students' abilities to 
plan solution strategies and implement them in problem settings that contain more elements and may be more 
'original' (or unfamiliar) than those in the connections cluster. In addition to the processes, knowledge and skills 
described for the connections cluster, the reflection cluster includes the following: 

■ Thinking and reasoning: this involves posing questions ("How do I find...?", "Which mathematics are 
involved...?", "What are the essential aspects of the problem or situation...?") and understanding the 
corresponding kinds of answers (provided by tables, graphs, algebra, figures, specification of key points, etc.); 
distinguishing between definitions, theorems, conjectures, hypotheses and assertions about special cases, 
and reflecting upon or actively articulating these distinctions; understanding and handling mathematical 
concepts in contexts that are new or complex; and understanding and handling the extent and limits of given 
mathematical concepts, and generalising results. 

m Argumentation: this involves simple mathematical reasoning, including distinguishing between proving 
and proofs and broader forms of argument and reasoning; following, assessing and constructing chains of 
mathematical arguments of different types; and using heuristics (e.g. What can or cannot happen, or be the 
case, and why?", "What do I know, and what do I want to obtain?'', "Which properties are essential?", "How 
are the objects related?"). 

■ Communication: this involves understanding and expressing oneself orally and in writing about mathematical 
matters ranging from reproducing the names and basic properties of familiar objects, and explaining 
computations and their results (usually in more than one way), to explaining matters that include complex 
relationships, including logical relationships. It also involves understanding others' written or oral statements 
about such matters. 

■ Modelling: this involves structuring the field or situation to be modelled; translating reality into mathematical 
structures in contexts that may be complex or largely different from what students are usually familiar with; 
interpreting back and forth between models (and their results) and reality, including aspects of communication 
about model results: gathering information and data, monitoring the modelling process and validating the 
resulting model. It also includes reflecting through analysing, offering a critique, and engaging in more 
complex communication about models and modelling. 

■ Problem posing and solving: this involves posing and formulating problems well beyond the reproduction of 
practised standard pure and applied problems in closed form; solving such problems by invoking and using 
standard approaches and procedures, but also more original problem-solving processes in which connections 
are being made between different mathematical areas and modes of representation and communication 
(schemata, tables, graphs, words, pictures). It also involves reflecting on strategies and solutions. 

■ Representation: this involves decoding, encoding and interpreting familiar and less familiar representations 
of mathematical objects; choosing and switching between different forms of representation of mathematical 
objects and situations, and translating and distinguishing between different forms of representation. It further 
involves the creative combination of representations and the invention of non-standard ones. 

■ Using symbolic, formal and technical language and operations: this involves decoding and interpreting 
symbolic and formal language practised in unknown contexts and situations, and handling statements and 
expressions containing symbols and formulae, including using variables, solving equations and undertaking 
calculations. It also involves the ability to deal with complex statements and expressions and with unfamiliar 
symbolic or formal language, and to understand and to translate between such language and natural language. 

■ Use of aids and tools: this involves knowing about and using familiar or unfamiliar aids and tools in contexts, 
situations and ways that are quite different from those in which their use was introduced and practised. It also 
involves knowing about limitations of aids and tools. 

Assessment items measuring the reflection cluster might be described with the following key descriptors: 
advanced reasoning, argumentation, abstraction, generalisation and modelling applied to new contexts. 



PISA 2009 ASSESSMENT FRAMEWORK - KEY COMPETENCIES IN READING, MATHEMATICS AND SCIENCE 0 OECD 2009 




PISA 2009 MATHEMATICS FRAMEWORK 



Examples of reflection cluster items 

MATHEMATICS EXAMPLE 21: STUDENT HEIGHTS 

In a mathematics class one day, the heights of all students were measured. The average height of boys was 
1 60 cm, and the average height of girls was 1 50 cm. Alena was the tallest - her height was 1 80 cm. Zdenek was 
the shortest - his height was 1 30 cm. 

Two students were absent from class that day, but they were in class the next day. Their heights were measured, 
and the averages were recalculated. Amazingly, the average height of the girls and the average height of the 
boys did not change. 

Question 1 : STUDENT HEIGHTS 

which of the followirig coriclusions cgn be 4 t 9 wn from this inform^tionf 
Circle 'Yes' or 'No' for egch coriclusion. 



Conclusion 


Can this conclusion be <\mwnl 


Both stu4ents gte qitls. 


Yes / No 


One of the stu4ents is g boy gn4 the other is g qirl. 


Yes / No 


Both stu4ents hgve the sgme height. 


Yes / No 


The gvergge height of gll stu4ents 4i4 not change. 


Yes / No 


Z4enek is still the shortest. 


Yes / No 



This problem is quite complicated in several ways. It requires very precise reading, as superficial reading will 
likely lead to misinterpretation. Norris and Phillips (2003) have argued that reading literacy is fundamental for 
science literacy. In a similar way, mathematical literacy also depends to some degree on reading skills. The 
item above demonstrates the importance of the receptive aspect of the communication competency, which is 
demanded strongly in this case. It also demonstrates how this aspect of mathematical literacy intersects with 
other types of literacy, specifically reading literacy. Furthermore, the wording of the problem makes it difficult 
to locate the crucial mathematical information. The intersection of other domains with the PISA definition 
and assessment of mathematics cannot be avoided; however, at the core of each assessment task there should 
be aspects that relate unambiguously to mathematical knowledge and skills. In this case, interpreting written 
information that contains mathematical data, and converting this information to a useful mathematical 
formulation are key mathematical challenges to be negotiated before a solution can be found. 

The situation varies within the class and over time. The entity class is used while discussing the average for boys and 
for girls independently, but subsequently it is stated that Alena is the tallest (girl or student) and Zdenek the shortest 
(boy or student). If the students do not read carefully they will miss the fact that Alena is a girl and Zdenek a boy. 

One obvious difficulty is the fact that the students have to combine the information from the first part of the 
stimulus (about the different heights) with the second part where the information about two missing students is 
presented. Here variation over time is seen: there are two students who were not present in the original setting, 
but have to be taken into account the next moment in time, so the entity class changes. However, the student 
solving the problem does not know whether the missing students are boys, girls or one of each. In addition, 
there is not one problem to solve, but five. 

Furthermore, to be able to answer correctly the students need to understand mathematically the statistical 
concepts involved. The problem involves the ability to pose questions ("How do I know...?", "How do I 
find...?", "What are the possibilities...?" and "What happens if I...?") and the ability to understand and handle 
the concept of an average in texts that are complex, although the context is familiar. 
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From this description it is clear that this item is not only challenging for students (as shown by the PISA results) 
but also clearly belongs to the reflection cluster. 

MATHEMATICS EXAMPLE 22: LIGHTHOUSE 

Lighthouses are towers with a light beacon on top. Lighthouses assist sea ships in finding their way 
at night when they are sailing close to the shore. 

A lighthouse beacon sends out light flashes with a regular fixed pattern. Every lighthouse has its 
own pattern. 

In the diagram below you see the pattern of a certain lighthouse. The light flashes alternate with 
dark periods. 



Light 



Dark 



0 1 2 3 4 5 6 7 8 9 10 11 12 13 

Time (sec) 

It is a regular pattern. After some time the pattern repeats itself. The time taken by one complete cycle of a 
pattern, before it starts to repeat, is called the period. When you find the period of a pattern, it is easy to extend 
the diagram for the next seconds or minutes or even hours. 

Question 1 : LIGHTHOUSE 

In the below, mgke 3 gt^pb of g possible pattern of light flashes of g lighthouse thgt sends out 

light flashes for 30 seconds pet minute. The period of this pattern must be egugl to 6 seconds. 





Light 



Dark 



0 1 2 3 4 . 5 6 7 8 9 10 11 12 13 

Time fsec) 

In this example, the students must first understand the introduction in the sense that this kind of graphs is 
most likely unknown to them, as is the idea of periodicity. In addition, the question posed is of a very open 
nature: the students are asked to design a possible pattern of light flashes. Many students do not encounter 
this kind of constructive question at school. However, this constructive aspect is an essential component of 
being mathematically literate: using mathematical competencies not only in a passive or derived way, but in 
constructing an answer. Solving the problem demands satisfying two conditions: equal amounts of time light 
and dark ("30 seconds per minute"), and a period of six seconds. This combination makes it essential for the 
students to engage with periodicity at the conceptual level - this involves the reflection cluster. 
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In this particular example, the context could be said to favour students living close to an ocean. It should be 
pointed out, however, that the PISA assessment of mathematics requires the capacity to use mathematics in 
contexts different from the local one. This ability to transfer is an essential competency. While certain students 
can be in a somewhat favourable position in certain contexts, and others in other contexts, the item by country 
analysis gives no indication that this is the case: landlocked countries did not perform differently from countries 
bordering on oceans. 

Classification of items by competency cluster 

Figure 2.9 summarises the distinctions between the clusters. 



Figure 2.9 ■ Diagram representing the competency clusters 




It is possible to use the descriptions outlined previously to classify mathematics items and thereby to assign them 
to one of the competency clusters. One way to do this is to analyse the demands of the item, then to rate each 
of the eight cognitive mathematical competencies for that item, according to which of the three clusters provide 
the most fitting description of item demands in relation to that competency. If any of the competencies were 
rated as fitting the description for the reflection cluster, then the item would be assigned to the reflection cluster. 
If not, but one or more of the competencies were rated as fitting the description for the connections cluster, then 
the item would be assigned to that cluster. Otherwise, the item would be assigned to the reproduction cluster, 
since all cognitive mathematical competencies would have been rated as fitting the descriptions for that cluster. 
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ASSESSING MATHEMATICS IN PISA 

Task characteristics 

This section considers, in more detail, features of the assessment tasks that are used to assess students. The 
nature of the tasks and the item format types are described below. 

The nature of PISA mathematics tasks 

PISA is an international survey of the knowledge and skills of 1 5-year-old students. All test items used should 
be suitable for the population of 1 5-year-old students in OECD countries. 

Trained markers have access to items including some stimulus material or information, an introduction, the 
actual question and the required solution. In addition, for items with responses that cannot be automatically 
coded, a detailed coding scheme is developed to enable trained markers across the range of participating 
countries to code the student responses in a consistent and reliable way. 

In an earlier section of this framework, the situations to be used for PISA mathematics items were discussed 
in some detail. For PISA, each item is set in one of four situation types: personal, educational/occupational, 
public and scientific. The items selected for the PISA mathematics instruments represent a spread across these 
situation types. 

In addition, item contexts that can be regarded as authentic are preferred. That is, PISA values most highly the tasks 
that could be encountered in real-world situations and have a context that demands the use of mathematics to 
solve the problem in an authentic manner. Problems with extra-mathematical contexts that influence the solution 
and its interpretation are preferred as vehicles for assessing mathematical processes, knowledge and skills. 

The selection of mathematics test items aims to ensure that the four overarching ideas are well represented. 
This framework requires the same of its mathematics test items. Items should relate predominantly to the 
overarching ideas (the phenomenological problem categories) described in the framework. Additionally, they 
should embody one or more of the mathematical processes that are described in the framework, and should be 
identified predominantly with one of the competency clusters. 

The level of reading required to successfully engage with an item is considered very carefully when items are 
developed and selected for inclusion in the PISA test instrument. The wording of items is as simple and direct 
as possible. Care is also taken to avoid question contexts that would create a cultural bias. 

Items selected for inclusion in the PISA test instruments represent a broad range of difficulties, to match the 
expected wide ability range of students participating in the PISA assessment. In addition, the major classifications 
of the framework (particularly competency clusters and overarching ideas) should, as much as possible, be 
represented by items of a wide range of difficulties. Item difficulties are established in an extensive field trial of 
test items prior to item selection for the main PISA survey. 

Item types 

When assessment instruments are devised, the impact of the item format type on student performance, and 
hence on the definition of the construct that is being assessed, must be carefully considered. This issue is 
particularly pertinent in a project such as PISA, where the large-scale cross-national context for testing puts 
serious constraints on the range of feasible item format types. 

The PISA mathematics assessment employs a combination of items with open constructed-response types, 
closed constructed-response types and multiple-choice types. About equal numbers of each of these item 
format types are used in constructing the test instruments. 

Based on experience in developing and using test items for PISA 2000, the multiple-choice type is generally 
regarded as most suitable for assessing items that would be associated with the reproduction and connections 
clusters. For an example of this item type, see Example 23, which shows an item that would be associated with 
the connections cluster and with a limited number of defined response options. To solve this problem, students 
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must translate the problem into mathematical terms, devise a model to represent the periodic nature of the 
context described, and extend the pattern to match the result with one of the given options. 

MATHEMATICS EXAMPLE 23: SEAL 

A seal has to breathe even if it is asleep. Martin observed a seal for one hour. At the start of his observation the 
seal dived to the bottom of the sea and started to sleep. In 8 minutes it slowly floated to the surface and took 
a breath. 

In 3 minutes it was back at the bottom of the sea again and the whole process started over in a very regular way. 

Question 1: SEAL 

After one hour the segl wgs; 

A. At the bottom 

B. On its wgy up 

C. Breathing 

P. On its w3y down 

Other format types are often preferred for some of the higher-order goals and more complex processes. Closed 
constructed-response items can pose similar questions to multiple-choice items, but students are asked to 
produce a response that can be easily judged as either correct or incorrect. For items in this type, guessing is 
not likely to be a concern, and the provision of distractors (which influence the construct that is being assessed) 
is not necessary. For example, the problem in Example 24 has one correct answer and many possible incorrect 
answers. 

MATHEMATICS EXAMPLE 24: FARMS 

Here you see a photograph of a farmhouse with a roof in the shape of a pyramid. 
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Below is a student's mathematical model of the farmhouse roof with measurements added. 




The attic floor, ABCD in the model, is a square. The beams that support the roof are the edges of a block 
(rectangular prism) EFGHKLMN. E is the middle of AT, F is the middle of BT, G is the middle of CT and H is the 
middle of DT. All the edges of the pyramid in the model have length 1 2 m. 

Question 1: FARMS 

Qlculgte the gteg ofthe gttic floor ABCP. 

The greg of the gttic floor ABCP = 

Open constructed-response items require a more extended response from the student, and the process of 
producing a response frequently involves higher-order cognitive activities. Often such items not only ask the 
student to produce a response, but also require the student to show their steps taken or explain how the answer 
was reached. The key feature of open constructed-response items is that students are allowed to demonstrate 
their abilities by providing solutions at a range of levels of mathematical complexity. 

Example 25 was not used in any PISA assessment, it exemplifies the kinds of features that are present in open 
constructed-response items. 

MATHEMATICS EXAMPLE 25: INDONESIA 



Indonesia lies between Malaysia and Australia. Some data of the population of Indonesia and its distribution 
over the islands is shown in the following table: 



Region 


Surface area (km ) 


Percentage of total 
area 


Population in 1980 
(millions) 


Percentage of total 
population 


Java/Madura 


132 187 


6.95 


91 281 


61.87 


Sumatra 


473 606 


24.86 


27 981 


18.99 


Kalimantan (Borneo) 


539 460 


28.32 


6 721 


4.56 


Sulawesi (Celebes) 


189 216 


9.93 


10377 


7.04 


Bali 


5 561 


0.30 


2 470 


1.68 


Irian jaya 


421 981 


22.16 


1 145 


5.02 


TOTAL 


1 905 569 


100.00 


147 384 


100.00 



One ofthe main challenges for Indonesia is the uneven distribution ofthe population over the islands. From the 
table we can see that Java, which has less than 7% of the total area, has almost 62% of the population. 

Source: de Lange and Verhage (1992). Used with permission. 
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Question 1: INDONESIA 

Pesign g gfgpb Cof gfgpbs) tbgt shows the uneven cjistfibution ofthe Indonesian population. 
The following item is an example of an open constructed-response item that was used in PISA 2003: 

MATHEMATICS EXAMPLE 26: WALKING 





The picture shows the footprints of a man walking. The pacelength P is the distance between the rear of two 
consecutive footprints. 



For men, the formula, ' 



n 



= 140 



, gives an approximate relationship between n and P where. 



n = number of steps per minute, and 



P = pace length in metres. 

Question 1 : WALKING 

Betngtci knows bis pgcelengtb is 0.80 metres. The fotmuU applies to Bernard's walking. 

Cgicuigte Bernard's walking speed in metres per minute gnd in kilometres per hour. Show your 
working out. 

About one-third ofthe mathematics items for PISA are open constructed-response items. The responses to these 
items require coding by trained individuals who implement a coding rubric that may require an element of 
professional judgement. Because of the potential for disagreement between coders of responses to these items, 
PISA implements coder reliability studies to monitor the extent of disagreement. Experience in these types of 
studies shows that clear coding rubrics can be developed and reliable scores can be obtained. 



PISA makes some use of a unit format in which several items are linked to common stimulus material. Tasks of 
this format give students the opportunity to become involved with a context or problem by asking a series of 
questions of increasing complexity. The first few questions are typically multiple-choice or closed constructed- 
response items, while subsequent ones are typically open constructed-response items. This format can be used 
to assess each of the three clusters. 



One reason for the use of common stimulus task formats is that they allow realistic tasks to be devised and the 
complexity of real-life situations to be reflected in them. Another reason relates to the efficient use of testing 
time, cutting down on the time required for a student to understand the problem. The need to make each scored 
point independent of others within a task is recognised and taken into account in the design of PISA tasks and 
the response coding and scoring rubrics. The importance of minimising bias that may result from using fewer 
situations is also recognised. 



Assessment structure 

In PISA 2003, when mathematics was the major PISA assessment domain, the test instruments contained a total 
of 21 0 minutes of testing time. The selected test items were arranged in seven clusters of items, with each item 
cluster representing 30 minutes of testing time, and the item clusters were placed in test booklets according to 
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a rotated test design. For the 2006 test cycle, when science was the major PISA assessment domain, less time 
was devoted to testing of mathematics, but the item clusters allocated to mathematics were constructed and 
rotated in a similar way. Similar arrangements apply to the mathematics test material for the PISA 2009 survey. 
In addition for 2009, clusters have been selected intact from those used in the 2006 survey to minimise any 
changed measurement effects arising from item placement factors. 

The total testing time for mathematics is distributed as evenly as possible across the four overarching ideas 
{space and shape, change and relationships, quantity and uncertainty) and the four situations described in the 
framework {personal, educational/occupational, public and scientific). The proportion of items reflecting the 
three clusters (reproduction, connections and reflection) is about 1:2:1, respectively. Multiple-choice response 
types, closed constructed-response types, and open constructed-response types each represent about one-third 
of the items. 

Aids and tools 

PISA policy allows students to use calculators and other tools as they are normally used in school. 

This represents the most authentic assessment of what students can achieve, and provides the most informative 
comparison of the performance of education systems. A system's choice to allow students to access and use 
calculators is no different, in principle, from other instructional policy decisions that are made by systems and 
are not controlled for by PISA. 

Students who are used to having a calculator available to assist them in answering questions would be 
disadvantaged if this resource were taken away. 



REPORTING PROFICIENCY IN MATHEMATICS 

To summarise data from responses to the PISA test instruments, a six-level described performance scale was 
created (Masters and Forster, 1996; Masters, Adams, and Wilson, 1999). Statistically created, the scale uses 
an item response modelling approach to scale ordered outcome data. The overall scale is used to describe 
the nature of performance by classifying the student performances of different countries in terms of the five 
described performance levels and thus providing a frame of reference for international comparisons. 

For the reporting of results from PISA 2003 when mathematics was the major domain, consideration was given 
to the development of a number of separate reporting scales. Such subscales could most obviously be based on 
the three clusters or the four overarching ideas. Decisions about the development of separate reporting scales 
were made on a variety of grounds, including psychometric considerations, following analysis of the data 
generated by the PISA assessments. To facilitate these possibilities, it was necessary to ensure that sufficient 
items were selected for inclusion in the PISA test instrument from each potential reporting category. Moreover, 
items within each such category needed to have a suitably wide range of difficulties. The balance of items 
across these categories has been broadly maintained in subsequent PISA administrations, but student outcomes 
have not been reported according to the content-based subscales for mathematics when it was a minor domain, 
such as in 2006 and 2009. 

The competency clusters described earlier in this framework reflect conceptual categories of broadly increasing 
cognitive demand and complexity, but do not strictly reflect a hierarchy of student performances based on item 
difficulty. Conceptual complexity is only one component of item difficulty that influences levels of performance. 
Others include familiarity, recent opportunity to learn and practice, etc. Thus, a multiple-choice item involving 
the reproduction cluster (for example, "Which of the following is a rectangular parallelepiped?" followed by 
pictures of a ball, a can, a box, and a square) may be very easy for students who have been taught the meaning 
of these terms, but very difficult for others because of their lack of familiarity with the terminology used. While 
it is possible to imagine relatively difficult reproduction cluster items and relatively easy reflection cluster items, 
one would expect a broadly positive relationship between competency clusters and item difficulty. 
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Factors that underpin increasing levels of item difficulty and mathematical proficiency include the following: 

■ The kind and degree of interpretation and reflection required: this includes the nature of demands arising 
from the problem context; the extent to which the mathematical demands of the problem are apparent or to 
which students must impose their own mathematical construction on the problem, and the extent to which 
insight, complex reasoning and generalisation are required. 

■ The kind of representation skills required: these include problems where only one mode of representation is 
used, as well as problems where students have to switch between different modes of representation or search 
for appropriate modes of representation themselves. 

■ The kind and level of mathematical skill required: these include single-step problems requiring students 
to reproduce basic mathematical facts and perform simple computation processes, multi-step problems 
involving more advanced mathematical knowledge, complex decision-making, information processing, and 
problem solving and modelling skills. 

■ The kind and degree of mathematical argumentation required: these include problems where no arguing is 
necessary, problems where students may apply well-known arguments, and problems where students have to 
create mathematical arguments, understand other people's argumentation or judge the correctness of given 
arguments or proofs. 

At the lowest described proficiency level, students typically carry out single-step processes that involve the 
recognition of familiar contexts and mathematically well-formulated problems, reproducing well-known 
mathematical facts or processes, and applying simple computational skills. 

At higher proficiency levels, students typically carry out more complex tasks involving more than a single 
processing step. They also combine different pieces of information or interpret different representations of 
mathematical concepts or information, recognising which elements are relevant and important and how they 
relate to one another. They typically work with given mathematical models or formulations, frequently in 
algebraic form, to identify solutions, or they carry out a small sequence of processing or calculation steps to 
produce a solution. 

At the highest proficiency level, students take a more creative and active role in their approach to mathematical 
problems. They typically interpret more complex information and negotiate a number of processing steps. They 
produce a formulation of a problem and often develop a suitable model that facilitates its solution. Students at 
this level can typically identify and apply relevant tools and knowledge in an unfamiliar problem context. They 
likewise demonstrate insight in identifying a suitable solution strategy, and display other higher-order cognitive 
processes such as generalisation, reasoning and argumentation to explain or communicate results. 

In Figure 2.10 the six described levels of mathematical proficiency developed for the PISA 2003 survey 
administration are presented, together with their associated scores on the PISA mathematics scale. The same 
descriptions were used in reporting the 2006 survey outcomes, and will also be used to report the PISA 2009 
outcomes. 
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Figure 2.10 ■ Summary descriptions of the six proficiency levels in mathematics 





Lower 

score 

limit 


What students can typically do at each level 


Level 6 


669.3 


At Level 6 students can conceptualise, generalise, and utilise information based on their 
investigations and modelling of complex problem situations. They can link different 
information sources and representations and flexibly translate among them. Students at this 
level are capable of advanced mathematical thinking and reasoning. These students can apply 
this insight and understandings along with a mastery of symbolic and formal mathematical 
operations and relationships to develop new approaches and strategies for attacking novel 
situations. Students at this level can formulate and precisely communicate their actions and 
reflections regarding their findings, interpretations, arguments, and the appropriateness of 
these to the original situations. 


Level 5 


607.0 


At Level 5 students can develop and work with models for complex situations, identifying 
constraints and specifying assumptions. They can select, compare, and evaluate appropriate 
problem solving strategies for dealing with complex problems related to these models. 
Students at this level can work strategically using broad, well-developed thinking and 
reasoning skills, appropriate linked representations, symbolic and formal characterisations, 
and insight pertaining to these situations. They can reflect on their actions and formulate and 
communicate their interpretations and reasoning. 


1 

i Level 4 


544.7 


At Level 4 students can work effectively with explicit models for complex concrete situations 
that may involve constraints or call for making assumptions. They can select and integrate 
different representations, including symbolic ones, linking them directly to aspects of real- 
world situations. Students at this level can utilise well-developed skills and reason flexibly, 
with some insight, in these contexts. They can construct and communicate explanations and 
arguments based on their interpretations, arguments, and actions. 


Level 3 


482.4 


At Level 3 students can execute clearly described procedures, including those that require 
sequential decisions. They can select and apply simple problem solving strategies. Students 
at this level can interpret and use representations based on different information sources 
and reason directly from them. They can develop short communications reporting their 
interpretations, results and reasoning. 


1 

Level 2 


420.1 


At Level 2 students can interpret and recognise situations in contexts that require no more 
than direct inference. They can extract relevant information from a single source and make 
use of a single representational mode. Students at this level can employ basic algorithms, 
formulae, procedures, or conventions. They are capable of direct reasoning and making literal 
interpretations of the results. 


Level 1 


357.8 


At Level 1 students can answer questions involving familiar contexts where all relevant 
information is present and the questions are clearly defined. They are able to identify 
information and to carry out routine procedures according to direct instructions in explicit 
situations. They can perform actions that are obvious and follow immediately from the given 
stimuli. 



CONCLUSION 

The aim of the PISA study, with regard to mathematics, is to develop indicators that show, from the perspective 
of the use of mathematics, how effectively countries have prepared their 1 5-year-old students to become active, 
reflective and intelligent citizens. To achieve this, PISA has developed assessments that focus on determining 
the extent to which students can use what they have learned. PISA emphasises mathematical processes, 
knowledge and understanding to solve problems that arise out of day-to-day experience, and provides a variety 
of problems with varying degrees of built-in guidance and structure, while simultaneously pushing towards 
authentic problems where students must do the thinking themselves. 
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This chapter presents the theory underlying the PISA 2009 science 
assessment. It begins with a definition of scientific literacy, outlines 
the organisation of science in PISA and sets the context for the 
test questions. The chapter describes the knowledge and skills at 
the heart of the assessment: identifying scientific issues, explaining 
phenomena scientifically and using scientific evidence. It then 
describes how knowledge and attitudes are also encompassed in 
the PISA definition of scientific literacy. Test questions are given 
as examples throughout this chapter to illustrate the classification, 
format and structure of the PISA science assessment. 
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INTRODUCTION 

This framework describes and illustrates the definition oi scientific literacy as used in PISA and sets the context 
for the items. Science is a minor domain in PISA 2009. The definition of the domain is unchanged since PISA 
2006 when for the first time it was the major domain assessed (OECD, 2006; Bybee & McCrae, 2009), though 
there are some changes in terminology, which have been brought about by an attempt to better align the 
language used in PISA with the language used in the OECD's DeSeCo initiative (OECD, 2003). 

In this framework, the term science literacy denotes an overarching competency comprising a set of three specific 
scientific competencies. A competency is more than just knowledge and skills (OECD, 2003). It includes the capacity 
to mobilise cognitive and non-cognitive resources in any given context. When discussing the cognitive dimensions 
of the specific scientific competencies, as is pertinent to the PISA science assessment in the current cycle, reference 
is made to the relevant scientific knowledge and skills demonstrated by students. However, the sub-scales of the PISA 
science scale as established in PISA 2006 (OECD, 2006) are still referred to as scientific competencies. 

In keeping with its status as a minor domain in this cycle, the student questionnaire will not include items asking 
about students' general attitudes towards science; nor will the main assessment instrument include questions on 
attitudes alongside the testing of cognitive abilities and knowledge, as was the case in PISA 2006. In this revised 
version of the science framework for PISA 2009, the section describing the PISA science assessment has been 
revised to reflect these changes, the discussion on reporting scales has been updated, and released examples 
from PISA 2006 have been included to illustrate the framework. 

An understanding of science and technology is central to a young person's preparedness for life in modern 
society. It enables an individual to participate fully in a society in which science and technology play a 
significant role. This understanding also empowers individuals to participate appropriately in the determination 
of public policy where issues of science and technology impact on their lives. An understanding of science and 
technology contributes significantly to the personal, social, professional and cultural lives of everyone. 

A large proportion of the situations, problems and issues encountered by individuals in their daily lives require 
some understanding of science and technology before they can be fully understood or addressed. Science 
and technology related issues confront individuals at personal, community, national and even global levels. 
Therefore, national leaders should be encouraged to ask about the degree to which all individuals in their 
respective countries are prepared to deal with these issues. A critical aspect of this is how young people respond 
to scientific questions when they emerge from school. An assessment at age 1 5 provides an early indication of 
how students may respond later in life to the diverse array of situations that involve science and technology. 

As the basis for an international assessment of 1 5-year-old students, it seems reasonable, therefore, to ask: What 
is it important for citizens to know, value, and be able to do in situations involving science and technology?" 
Answering this question establishes the basis for an assessment of students with regards to how their knowledge, 
values and abilities today relate to what they will need in the future. Central to the answer are the competencies 
that lie at the heart of the PISA science assessment. These ask how well students: 

■ identify scientific issues 

■ explain phenomena scientifically 

■ use scientific evidence 

These competencies require students to demonstrate, on the one hand, knowledge and cognitive abilities, and 
on the other, attitudes, values and motivations, as they meet and respond to science-related issues. 

The issue of identifying what citizens should know, value and be able to do in situations involving science 
and technology, seems simple and direct. However, doing so raises questions about scientific understanding 
and does not imply mastery of all scientific knowledge. This framework is guided by reference to what citizens 
require. As citizens, what knowledge is most appropriate? An answer to this question certainly includes basic 
concepts of the science disciplines, but that knowledge must be used in contexts that individuals encounter in 
life. In addition, people often encounter situations that require some understanding of science as a process that 
produces knowledge and proposes explanations about the natural world.’ Further, they should be aware of the 
complementary relationships between science and technology, and how science-based technologies pervade 
and influence the nature of modern life. 
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What is important for citizens to value about science and technology? An answer should include the role 
and contributions to society of science and of science-based technology, and their importance in many 
personal, social, and global contexts. Accordingly, it seems reasonable to expect individuals to have an 
interest in science, to support the process of scientific enquiry and to act responsibly towards natural 
resources and the environment. 

What is important for individuals to be able to do that is science-related? People often have to draw appropriate 
conclusions from evidence and information given to them; they have to evaluate claims made by others on 
the basis of the evidence put forward and they have to distinguish personal opinion from evidence-based 
statements. Often the evidence involved is scientific, but science has a more general role to play as well since 
it is concerned with rationality in testing ideas and theories against evidence. Of course this does not deny 
that science includes creativity and imagination, attributes that have always played a central part in advancing 
human understanding of the world. 

Can citizens distinguish claims that are scientifically sound from those that are not? Ordinary citizens are 
generally not called on to judge the worth of major theories or potential advances in science. But they do make 
decisions based on the facts in advertisements, evidence in legal matters, information about their health, and 
issues concerning local environments and natural resources. An educated person should be able to distinguish 
the kinds of questions that can be answered by scientists and the kinds of problems that can be solved by 
science-based technologies from those that cannot be answered in these ways. 



DEFINING THE DOMAIN 

Current thinking about the desired outcomes of science education emphasises scientific knowledge (including 
knowledge of the scientific approach to enquiry) and an appreciation of science's contribution to society. These 
outcomes require an understanding of important concepts and explanations of science, and of the strengths 
and limitations of science in the world. They imply a critical stance and a reflective approach to science 
(Millar & Osborne, 1998). 

Such goals provide an orientation and emphasis for the science education of all people (Fensham, 1985). The 
competencies assessed in PISA are broad and include aspects that relate to personal utility, social responsibility, 
and the intrinsic and extrinsic value of scientific knowledge. 

The above discussion frames a central point of the PISA science assessment: the assessment should focus on 
scientific competencies that clarify what 1 5-year-old students know, value and are able to do within reasonable 
and appropriate personal, social and global contexts. This perspective differs from one grounded exclusively in 
school science programmes and extensively based only on the disciplines of science; but it includes problems 
situated in educational contexts and also in professional ones, and recognises the essential place of the 
knowledge, methods, attitudes, and values that define scientific disciplines (Bybee, 1997b; Fensham, 2000; 
Craber & Bolte, 1997; Mayer, 2002; Roberts, 1983; UNESCO, 1993). 

PISA is concerned with both the cognitive and affective aspects of students' competencies in science. The 
cognitive aspects include students' knowledge and their capacity to use this knowledge effectively, as they carry 
out certain cognitive processes that are characteristic of science and scientific enquiries of personal, social, 
or global relevance. In assessing scientific competencies, PISA is concerned with issues to which scientific 
knowledge can contribute and which will involve students, either now or in the future, in making decisions. 
From the point of view of their scientific competencies, students respond to such issues in terms of their 
understanding of relevant scientific knowledge, their ability to access and evaluate information, their ability to 
interpret evidence bearing on the issue and their ability to identify the scientific and technological aspects of 
the issue (Koballa, Kemp & Evans, 1997; Law, 2002). PISA also is concerned with non-cognitive aspects: how 
students respond affectively. Attitudinal aspects of their response engage their interest, sustain their support, and 
motivate them to take action (Schibeci, 1984). 
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Box 3.1 ■ Scientific knowledge: PISA terminology 



The term scientific knowledge is used throughout this framework to refer to both knowledge of science 
and knowledge about science. Knowledge of science refers to knowledge of the natural world across 
the major fields of physics, chemistry, biological science. Earth and space science, and science-based 
technology. Knowledge about science refers to knowledge of the means (scientific enquiry) and goals 
(scientific explanations) of science. 



The PISA science assessment encompasses a continuum of scientific knowledge and the cognitive abilities 
associated with scientific enquiry, incorporates multiple dimensions, and addresses the relationships between 
science and technology. It provides an assessment of students' scientific literacy by assessing their capacity to 
use scientific knowledge (Bybee, 1997a; Fensham, 2000; Law, 2002; Mayer & Kumano, 2002). 



Box 3.2 b PISA scientific literacy 



For the purposes of PISA, scientific literacy refers to an individual's: 

• Scientific knowledge and use of that knowledge to identify questions, acquire new knowledge, explain 
scientific phenomena and draw evidence-based conclusions about science-related issues 

• Understanding of the characteristic features of science as a form of human knowledge and enquiry 

• Awareness of how science and technology shape our material, intellectual and cultural environments 

• Willingness to engage in science-related issues, and with the ideas of science, as a reflective citizen 



Scientific literacy 

The following remarks clarify the definition of scientific literacy as defined for the purposes of PISA. 

Using the term "scientific literacy" rather than "science" underscores the importance that the PISA science 
assessment places on the application of scientific knowledge in the context of life situations, compared with 
the simple reproduction of traditional school science knowledge. The functional use of knowledge requires 
the application of those processes that are characteristic of science and scientific enquiry (here termed the 
scientific competencies) and is regulated by the individual's appreciation, interest, values, and action relative 
to scientific matters. A student's ability to carry out the scientific competencies involves both knowledge of 
science and an understanding of the characteristics of science as a way of acquiring knowledge (/.e. knowledge 
about science). The definition also recognises that the disposition to carry out these competencies depends 
upon an individual's attitudes towards science and a willingness to engage in science-related issues. 

Knowledge and use of that knowledge to identify questions, to acquire new knowledge, to 
explain scientific phenomena and to draw evidence-based conclusions 

Knowledge for this definition implies far more than the ability to recall information, facts, and names. The definition 
includes knowledge of science (knowledge about the natural world) and knowledge about science itself. Knowledge 
of science includes understanding fundamental scientific concepts and theories; knowledge about science includes 
understanding the nature of science as a human activity and the power and limitations of scientific knowledge. The 
questions to be identified are those that can be answered by scientific enquiry, again requiring knowledge about 
science as well as scientific knowledge of the specific topics involved. Of significance is that individuals must often 
acquire new knowledge not through their own scientific investigations, but through resources such as libraries and 
the Internet. Drawing evidence-based conclusions means knowing, selecting and evaluating information and data. 
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while recognising that there is often not sufficient information to draw definite conclusions, thus making it necessary 
to speculate cautiously and consciously about the information that is available. 

Characteristic features of science as a form of human knowledge and enquiry 

As expressed here, being scientifically literate implies that students should have some understanding of how 
scientists obtain data and propose explanations, and recognise key features of scientific investigations and 
the types of answers one can reasonably expect from science. For example, scientists use observations and 
experiments to gather data about objects, organisms and events in the natural world. The data are used to 
propose explanations that become public knowledge and may be used in various forms of human activity. 
Some key features of science include: the collection and use of data - data collection is guided by ideas 
and concepts (sometimes stated as hypotheses) and includes issues of relevance, context and accuracy; the 
tentative nature of knowledge claims; an openness to sceptical review; the use of logical arguments; and, 
the obligation to make connections to current and historical knowledge, and to report the methods and 
procedures used in obtaining evidence. 

How science and technology shape our material, intellectual, and cultural environments 

The key points in this statement include the idea that science is a human endeavour, one that influences our 
societies and us as individuals. Further, technological development is also a human endeavour (Fleming, 1 989). 
Although science and technology differ in aspects of their purposes, processes, and products, they are also 
closely related and, in many respects, complementary. In this regard, the definition of scientific literacy as used 
here includes the nature of science and of technology and their complementary relationships. As individuals 
we make decisions through public policies that influence the directions of science and technology. Science 
and technology play paradoxical roles in society as they propose answers to questions and provide solutions to 
problems, but may also create new questions and problems. 

Willingness to engage in science-related issues and with the ideas of science as a reflective 
citizen 

The meaning conveyed in the first part of this statement, 'willingness to engage in science-related issues', is 
wider than taking note and taking action as required; it implies having continuing interest in, having opinions 
about and participating in current and future science-based issues. The second part of the statement, 'with 
the ideas of science as a reflective citizen', covers various aspects of attitudes and values that individuals may 
have towards science. The whole phrase implies a person who has an interest in scientific topics, thinks about 
science-related issues, is concerned about issues of technology, resources and the environment, and reflects on 
the importance of science in personal and social perspectives. 

Inevitably, scientific competencies draw upon reading and mathematical competencies (Norris & Phillips, 2003). 
For example, aspects of mathematical competencies are required in data interpretation contexts. Similarly, reading 
literacy is necessary when a student is demonstrating an understanding of scientific terminology. The intersection 
of these other domains with the PISA definition and assessment of science cannot be avoided; however, at the core 
of each assessment task there should be aspects that relate unambiguously to science competency. 



ORGANISING THE DOMAIN 

The definition of the science domain proposed here provides for a continuum in which individuals are deemed to 
be more or less scientifically literate; they are not regarded as either scientifically literate or scientifically illiterate 
(Bybee, 1 997a; 1 997b). So, for example, the student with less developed scientific literacy might be able to recall 
simple scientific factual knowledge and to use common scientific knowledge in drawing or evaluating conclusions. 
A student with more developed scientific literacy will demonstrate the ability to create and use conceptual models 
to make predictions and give explanations, analyse scientific investigations, relate data as evidence, evaluate 
alternative explanations of the same phenomena, and communicate conclusions with precision. 
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For assessment purposes, the PISA definition of scientific literacy may be characterised as consisting of four 
interrelated aspects (see Figure 3.1): 

■ Context: recognising life situations involving science and technology. 

■ Knowledge: understanding the natural world on the basis of scientific knowledge that includes both 
knowledge of the natural world, and knowledge about science itself. 

■ Competencies: demonstrating scientific competencies that include identifying scientific issues, explaining 
phenomena scientifically, and drawing conclusions based on evidence. 

m Attitudes: indicating an interest in science, support for scientific enquiry, and motivation to act responsibly 
towards, for example, natural resources and environments. 

The following sections restate and elaborate these interrelated aspects. In highlighting these aspects, the PISA 
science framework has ensured that the focus of the assessment is upon the outcomes of science education. 
Several questions have guided the establishment of the PISA science framework. They are: 

■ What contexts would be appropriate for assessing 1 5-year-old students? 

■ What competencies might we reasonably expect 1 5-year-old students to demonstrate? 

■ What knowledge might we reasonably expect 1 5-year-old students to demonstrate? 

■ What attitudes might we reasonably expect 1 5-year-old students to demonstrate? 

Figure 3.1 ■ Framework for PISA science assessment 
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(See Figure 3.2) 
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(See Figure 3.3) 
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Situations and context 

An important aspect of the PISA science assessment is engagement with science in a variety of situations. In 
dealing with scientific issues, the choice of methods and representations is often dependent on the situations 
in which the issues are presented. 

The situation is the part of the student's world in which the tasks are placed. Assessment items are framed in 
situations of general life and not limited to life in school. In the PISA science assessment, the focus of the items 
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is on situations relating to the self, family and peer groups {personal), to the community (social) and to life 
across the world (global). A further type of setting, appropriate to some topics, is the historical one, in which 
understanding of the advances in scientific knowledge can be assessed. 

PISA assesses important scientific knowledge relevant to the science education curricula of participating countries 
without being constrained to the common aspects of participants' national curricula. The assessment does this 
by requiring evidence of the successful use of science knowledge and skills in important situations reflecting the 
world and in accordance with PISA's focus. This, in turn, involves the application of selected knowledge about the 
natural world, and about science itself, and evaluation of students' attitudes towards scientific matters. 

Figure 3.2 lists the main applications of science that are involved within personal, social, and global settings as the 
contexts, or specific situations, for assessment exercises. However, other settings (e.g. technological, historical) and 
areas of application are also used. The areas of application are: health , natural resources , the environment , 
hazards , and the frontiers of science and technology .They are the areas in which science has particular value for 
individuals and communities in enhancing and sustaining quality of life, and in the development of public policy. 

The PISA science assessment is not an assessment of contexts. It assesses competencies, knowledge and attitudes 
as these are presented or relate to contexts. In selecting the contexts, it is important to keep in mind that the 
purpose of the assessment is to assess scientific competencies, understandings, and attitudes that students have 
acquired by the end of the compulsory years of schooling. 

PISA items are arranged in groups (units) based around a common stimulus that establishes the context for 
the items. The contexts used are chosen in the light of relevance to students' interests and lives. The items are 
developed keeping in mind linguistic and cultural differences in participating countries. 

Figure 3.2 ■ Contexts for PISA science assessment 





Personal 

(self, family and peer groups) 


Social 

(the community) 


Global 

(life across the world) 


Health 


maintenance of health, 
accidents, nutrition 


control of disease, social 
transmission, food choices, 
community health 


epidemics, spread of infectious 
diseases 


Natural 

resources 


personal consumption of 
materials and energy 


maintenance of human 
populations, quality of life, 
security, production and 
distribution of food, energy 
supply 


renewable and non-renewable 
energy sources, natural systems, 
population growth, sustainable 
use of species 


Environment 


environmentally friendly 
behaviour, use and disposal of 
materials 


population distribution, 
disposal of waste , 
environmental impact, local 
weather 


biodiversity, ecological 
sustainability, control of 
pollution, production and loss 
of soil 


Hazard 


natural and human-induced, 
decisions about housing 


rapid changes (earthquakes, 
severe weather), slow and 
progressive changes (coastal 
erosion, sedimentation), risk 
assessment 


climate change, impact of 
modern warfare 


Frontiers of 
science and 
technology 


interest in science s 
explanations of natural 
phenomena, science-based 
hobbies, sport and leisure, 
music and personal technology 


new materials, devices and 
processes, genetic modification, 
weapons technology, transport 


extinction of species, 
exploration of space, origin and 
structure of the universe 



Science examples 

In this section, three examples of science units from the PISA 2006 assessment are presented. They are referred to 
throughout the remainder of the chapter to illustrate the variety of contexts involved, the scientific competencies 
and areas of scientific knowledge addressed by PISA science items, and the item types (formats) employed. In 
addition, the scoring guide for each item is shown (for a description of proficiency levels, see Figure 3.10). 
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Science example 1 

Science example 1 is a unit titled ACID RAIN. The stimulus material is a photograph of statues on the Acropolis 
in Athens, together with a brief statement explaining that the original statues were moved inside the museum 
of the Acropolis due to their deterioration from acid rain. The area of application is Hazards within personal 
and social settings. 

SCIENCE EXAMPLE 1 : ACID RAIN 

Below is a photo of statues called Caryatids that were built on the Acropolis in Athens more than 2500 years 
ago. The statues are made of a type of rock called marble. Marble is composed of calcium carbonate. 

In 1 980, the original statues were transferred inside the museum of the Acropolis and were replaced by replicas. 
The original statues were being eaten away by acid rain. 




Question 1 : ACID RAIN 

Notmgl tgin is slightly 3d4ic because it bgs absorbed some cgtbon 4ioxi4e from the git. Aci4 tgin is mote 
gd4ic tbgn notmgl tgin becguse it hgs gbsotbe4 ggses like sulfut oxi4es gn4 nitrogen oxi4es gs well. 

Wbete 4o these sulfut oxi4es gn4 nitrogen oxi4es in the git come ftomf 



Full Credit (Level 3: 506) 

Any one of car exhausts, factory emissions, burning fossil fuels such as oil and coal, gases from volcanoes or 
other similar things OR Responses that include an incorrect as well as a correct source of the pollution OR 
Responses that refer to "pollution" but do not give a source of pollution that is a significant cause of acid rain. 
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The effect of acid rain on marble can be modelled by placing chips of marble in vinegar overnight. Vinegar and 
acid rain have about the same acidity level. When a marble chip is placed in vinegar, bubbles of gas form. The 
mass of the dry marble chip can be found before and after the experiment. 

Question 2: ACID RAIN 

A mgtble chip hgs g m^ss of 2.0 gtgms before being immersed in vineggr overnight. The chip is 
removed 3nd dtied the next d^y. Whgt will the m^ss ofthe dtied mgrble chip bef 

A. Less thgn 2.0 grgms 

B. Exactly 2.0 grgms 

C. Between 2.0 gnd 2.4. grgms 

D. More thgn 2.4 grgms 

Full Credit (Level 2; 460) 

A. Less than 2.0 grams 

Question 3: ACID RAIN 

Students who did this experiment giso pUced mgrble chips in pure (distilled) wgter overnight. 

Explain why the students included this step in their experiment. 



Full Credit (Level 6; 717) 

To show that the acid (vinegar) is necessary for the reaction. 

Partial Credit (Level 3: 513) 

To compare with the test of vinegar and marble, but it is not made clear that this is being done to show that the 
acid (vinegar) is necessary for the reaction. 

Science example 2 

Science example 2 is titled GREENHOUSE and deals with the increase of the average temperature of the 
Earth's atmosphere. The stimulus material consists of a short text introducing the term "Greenhouse effect" and 
includes graphical information on the average temperature of the Earth's atmosphere and the carbon dioxide 
emission on the Earth over time. 

The area of application is Environment within a global setting. 

SCIENCE EXAMPLE 2: GREENHOUSE 

Read the texts and answer the questions that follow. 

THE GREENHOUSE EEEECT: EACT OR EICTION? 

Eiving things need energy to survive. The energy that sustains life on the Earth comes from the Sun, which 
radiates energy into space because it is so hot. A tiny proportion of this energy reaches the Earth. 

The Earth's atmosphere acts like a protective blanket over the surface of our planet, preventing the variations in 
temperature that would exist in an airless world. 
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Most of the radiated energy coming from the Sun passes through the Earth's atmosphere. The Earth absorbs 
some of this energy, and some is reflected back from the Earth's surface. Part of this reflected energy is absorbed 
by the atmosphere. 

As a result of this the average temperature above the Earth's surface is higher than it would be if there were no 
atmosphere. The Earth's atmosphere has the same effect as a greenhouse, hence the term greenhouse effect. 

The greenhouse effect is said to have become more pronounced during the twentieth century. 

It is a fact that the average temperature of the Earth's atmosphere has increased. In newspapers and periodicals 
the increased carbon dioxide emission is often stated as the main source of the temperature rise in the twentieth 
century. 

A student named Andr becomes interested in the possible relationship between the average temperature of the 
Earth's atmosphere and the carbon dioxide emission on the Earth. 

In a library he comes across the following two graphs. 



Carbon dioxide emission 
(thousand miiiions of 
tonnes per year) 



20 



T 



10 



I860 1870 1880 1890 1900 1910 1920 1930 1940 1950 1960 1970 1980 1990 




years 



Average temperature 
of the Earth's 
atmosphere (°C) 




years 



Andr concludes from these two graphs that it is certain that the increase in the average temperature of the 
Earth's atmosphere is due to the increase in the carbon dioxide emission. 
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Question 1 ; GREENHOUSE 

Whgt is it gbout the gfgphs thgt supports Andre's conclusion!’ 



Full Credit (Level 3: 529) 

Refers to the increase of both (average) temperature and carbon dioxide emission OR Refers (in general terms) 
to a positive relationship between temperature and carbon dioxide emission. 



Question 2: GREENHOUSE 

Another student, jegnne, disagrees with Andre's conclusion. She compares the two graphs gnd sgys 
thgt some pgrts ofthe graphs do not support his conclusion. 

Give gn example of g pgrt of the graphs thgt does not support Andre's conclusion. ExpUin your 
answer. 



Full Credit (Level 5; 659) 

Refers to one particular part of the graphs in which the curves are not both descending or both climbing and 
gives the corresponding explanation. 

Partial Credit (Level 4: 568) 

Mentions a correct period, without any explanation OR Mentions only one particular year (not a period of 
time), with an acceptable explanation OR Gives an example that doesn't support Andre's conclusion but makes 
a mistake in mentioning the period OR Refers to differences between the two curves, without mentioning a 
specific period OR Refers to an irregularity in one of the graphs OR Indicates difference in the graphs, but 
explanation is poor. 

Question 3; GREENHQUSE 

Aridre persists in his conclusion thgt the gvergge temperature rise of the Egrth's atmosphere is cgused 
by the increase in the cgrbon dioxide emission. But jegnne thinks thgt his conclusion is premature, 
she sgys; "Before accepting this conclusion you must be sure thgt other factors thgt could influence 
the greenhouse effect gre constant". 

Ngme one of the factors thgt jegnne megns. 



Full Credit (Level 6; 709) 

Gives a factor referring to the energy/radiation coming from the Sun OR Gives a factor referring to a natural 
component or a potential pollutant. 
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Science example 3 

Science example 3 titled PHYSICAL EXERCISE is concerned with the effect of physical exercise on personal 
health. 

SCIENCE EXAMPLE 3: PHYSICAL EXERCISE 



Regular but moderate physical exercise is good for our health. 




Question 1 : PHYSICAL EXERCISE 

Whgt gte the gdvgntgges of regulgr physical exerdsei’ Circle "Yes" or "No" for e^ch statement. 



Is this an a4vantage of regular physical exercise? 


Yes or No? 


Physical exercise helps prevent heart and circulation illnesses. 


Yes / No 


Physical exercise leads to a healthy diet. 


Yes / No 


Physical exercise helps to avoid becoming overweight. 


Yes / No 


Full Credit (Level 3: 545) 

All three correct: Yes, No, Yes in that order. 




Question 2: PHYSICAL EXERCISE 



Whgt happens when muscles gre exercisedi* Circle "Yes" or "No" for egcb statement. 



Does this happen when muscles are exercised? 


Yes or No? 


Muscles get an increased flow of blood. 


Yes / No 


Fats are formed in the muscles. 


Yes / No 



Full Credit (Level 1 ; 386) 

Both correct: Yes, No in that order. 
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Question 3; PHYSICAL EXERCISE 

why cjo you hgve to bfegthe mofe hegvily when you're doing physical exercise tbgn when youf body 
is resting!' 



Full Credit (Level 4; 583) 

To remove increased levels of carbon dioxide and to supply more oxygen to your body OR To remove increased 
levels of carbon dioxide from your body or to supply more oxygen to your body, but not both. 

Scientific competencies 

The PISA science assessment gives priority to the competencies listed in Figure 3.3: the ability to identify 
scientifically-oriented issues; describe, explain or predict phenomena based on scientific knowledge; interpret 
evidence and conclusions, and use scientific evidence to make and communicate decisions. Demonstrating 
these competencies in the PISA assessment involves applying scientific knowledge - both knowledge of science 
and knowledge about science itself as a form of knowledge and an approach to enquiry. 

Figure 5.5 ■ PISA scientific competencies 



■ Recognising issues that are possible to investigate scientifically 

■ Identifying keywords to search for scientific information 

■ Recognising the key features of a scientific investigation 



■ Applying knowledge of science in a given situation 

■ Describing or interpreting phenomena scientifically and predicting changes 

■ Identifying appropriate descriptions, explanations, and predictions 



■ Interpreting scientific evidence and making and communicating conclusions 

■ Identifying the assumptions, evidence and reasoning behind conclusions 

■ Reflecting on the societal implications of science and technological 
developments 



Some cognitive processes have special meaning and relevance for scientific literacy. Among the cognitive 
processes that are implied in the scientific competencies are: inductive reasoning (reasoning from detailed 
facts to general principles) and deductive reasoning (reasoning from the general to the particular), critical 
and integrated thinking, transforming representations (e.g. data to tables, tables to graphs), constructing and 
communicating arguments and explanations based on data, thinking in terms of models, and using mathematical 
processes, knowledge and skills. 
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Justification for an emphasis on the scientific competencies of Figure 3.3 in PISA rests on the importance of 
these competencies for scientific investigation. They are grounded in logic, reasoning, and critical analysis. An 
elaboration of the scientific competencies follows, including references to how they are assessed in the science 
examples presented in the previous section 

Identifying scientific issues 

It is important to be able to distinguish scientific issues and content from other forms of issues. Importantly, scientific 
issues must lend themselves to answers based on scientific evidence. The competency identifying scientific issues 
includes recognising questions that it would be possible to investigate scientifically in a given situation and identifying 
keywords to search for scientific information on a given topic. It also includes recognising key features of a scientific 
investigation: for example, what things should be compared, what variables should be changed or controlled, what 
additional information is needed, or what action should be taken so that relevant data can be collected. 

Identifying scientific issues requires students to possess knowledge about science itself, and may also draw, to 
varying degrees, on their knowledge of science. For example. Question 3 of ACID RAIN (Science example 1 ) 
requires students to answer a question about the control in a scientific investigation. Students must compare 
an acid (vinegar) reaction to possible reactions with pure water to be sure that acid is the cause of the reaction. 

Explaining phenomena scientifically 

Demonstrating the competency explaining phenomena scientifically involves applying appropriate knowledge 
of science in a given situation. The competency includes describing or interpreting phenomena and predicting 
changes, and may involve recognising or identifying appropriate descriptions, explanations, and predictions. 
An example of a PISA item that requires students to explain phenomena scientifically is Question 1 of ACID 
RAIN (Science example 1), where students must explain the origin of sulfur oxides and nitrogen oxides in the 
air. Other examples are: Question 3 of GREENHOUSE (Science example 2), which requires students to identify 
factors causing the average temperature rise of the Earth, and Question 3 of PHYSICAL EXERCISE (Science 
example 3) which asks students to apply their knowledge of the human respiratory system. 

Using scientific evidence 

The competency using scientific evidence includes accessing scientific information and producing arguments and 
conclusions based on scientific evidence (Kuhn, 1992; Osborne, Erduran, Simon & Monk, 2001). The required 
response can involve knowledge about science or knowledge of science or both. Question 2 of ACID RAIN (Science 
example 1) requires students to use the information provided to form a conclusion about the effects of vinegar 
on marble, a simple model for the influence of acid rain on marble. Other examples are Questions 1 and 2 of 
GREENHOUSE (Science example 2), both of which require students to interpret evidence presented in two graphs. 

The competency also involves: selecting from alternative conclusions in relation to evidence; giving reasons 
for or against a given conclusion in terms of the process by which the conclusion was derived from the data 
provided; and identifying the assumptions made in reaching a conclusion. Reflecting on the societal implications 
of scientific or technological developments is another aspect of this competency. 

Students may be required to express their evidence and decisions to a specified audience, through their own 
words, diagrams or other representations as appropriate. In short, students should be able to present clear and 
logical connections between evidence and conclusions or decisions. 

Scientific knowledge 

As previously noted (see Box 3.1), scientific knowledge refers to both knowledge of science (knowledge about 
the natural world) and knowledge about science itself. 

Knowledge of science 

Given that only a sample of students' knowledge of science can be assessed in any one PISA assessment, it 
is important that clear criteria are used to guide the selection of knowledge that is assessed. Moreover, the 
objective of PISA is to describe the extent to which students can apply their knowledge in contexts of relevance 
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to their lives. Accordingly, the assessed knowledge will be selected from the major fields of physics, chemistry, 
biology, Earth and space science, and technology according to the following criteria: 

■ The relevance to real-life situations: scientific knowledge differs in the degree to which it is useful in the life 
of individuals. 

■ The knowledge selected represents important scientific concepts and thus has enduring utility. 

■ The knowledge selected is appropriate to the developmental level of 1 5-year-old students. 

Figure 3.4 shows the knowledge of science categories and examples of content selected by applying these criteria. 
This knowledge is required for understanding the natural world and for making sense of experiences in personal, 
social and global situations. The framework uses the term systems instead of sciences in the descriptors of 
the major fields to convey the idea that citizens have to apply their understanding of concepts from the physical 
and life sciences. Earth and space science, and technology, in situations that interact in more or less united ways. 

Figure 5.4 ■ PISA categories of knowledge of science 



■ Structure of matter {e.g. particle model, bonds) 

■ Properties of matter {e.g. changes of state, thermal and electrical conductivity) 

■ Chemical changes of matter (e.g. reactions, energy transfer, acids/bases) 

■ Motions and forces (e.g. velocity, friction) 

■ Energy and its transformation (e.g. conservation, dissipation, chemical reactions) 

■ Interactions of energy and matter (e.g. light and radio waves, sound and seismic waves) 

■ Cells (e.g. structures and function, DNA, plant and animal) 

■ Humans (e.g. health, nutrition, subsystems \i.e. digestion, respiration, circulation, excretion, and their 
relationship], disease, reproduction) 

■ Populations (e.g. species, evolution, biodiversity, genetic variation) 

■ Ecosystems (e.g. food chains, matter and energy flow) 

■ Biosphere (e.g. ecosystem services, sustainability) 

■ Structures of Earth systems (e.g. lithosphere, atmosphere, hydrosphere) 

■ Energy in Earth systems (e.g. sources, global climate) 

■ Change in Earth systems (e.g. plate tectonics, geochemical cycles, constructive and destructive forces) 

■ Earth’s history (e.g. fossils, origin and evolution) 

■ Earth in space (e.g. gravity, solar systems) 

■ Role of science -based technology (e.g. solve problems, help humans meet needs and wants, design and 
conduct investigations) 

■ Relationships between science and technology (e.g. technologies contribute to scientific advancement) 

■ Concepts (e.g. optimisation, trade-offs, cost, risk, benefit) 

■ Important principles (e.g. criteria, constraints, innovation, invention, problem solving) 
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The examples listed in Figure3.4convey the meanings of the categories, 'there is no attempt to list comprehensively 
all the knowledge that could be related to each of the knowledge of science categories. 

Question 2 of ACID RAIN (Science example 1) assesses students' knowledge of science in the category "Physical 
systems . Question 3 oi ACID RAIN focuses on students' knowledge about science in the category "Scientific enquiry," 
but assumes some knowledge of science (category "Physical systems") that students can be expected to possess. 

Question 3 of GREENHOUSE (Science example 2) is concerned with students' knowledge of "Earth and space 
systems ; and Questions 1 , 2 and 3 of PHYSICAL EXERCISE (Science example 3) assess students' knowledge of 
"Living systems". 

Knowledge about science 

Figure 3.5 displays the categories and examples of content for knowledge about science. The first category, 
"Scientific enquiry," centres on enquiry as the central process of science and the various components of that 
process. The second category, closely related to enquiry, is "Scientific explanations". Scientific explanations 
are the results of scientific enquiry. One can think of enquiry as the means of science (how scientists get data) 
and explanations as the goals of science (how scientists use data). The examples listed in Figure 3.5 convey the 
general meanings of the categories; there is no attempt to list comprehensively all the knowledge that could be 
related to each category. 

Figure 3.5 ■ PISA categories of knowledge about science 



■ Origin (e.g. curiosity, scientific questions) 

■ Purpose (e.g. to produce evidence that helps answer scientific questions, current ideas/ models/ theories 
guide enquiries) 

■ Experiments (e.g. different questions suggest different scientific investigations, design) 

■ Data type (e.g. quantitative [measurements], qualitative [observations]) 

■ Measurement (e.g. inherent uncertainty, replicability, variation, accuracy/precision in equipment and 
procedures) 

■ Characteristics of results (e.g. empirical, tentative, testable, falsifiable, self-correcting) 



■ Types (e.g. hypothesis, theory, model, law) 

■ Formation (e.g. data representation, role of extant knowledge and new evidence, creativity and 
imagination , logic) 

■ Rules (e.g. must be logically consistent; based on evidence, historical and current knowledge) 

■ Outcomes (e.g. produce new knowledge, new methods, new technologies; lead to new questions and 
investigations) 



Question 3 of ACID RAIN (Science example 1) is an example of knowledge about science in the category 
"Scientific enquiry." The question requires students to identify the possible purposes for a control of an 
investigation (competency: Identifying Scientific Issues). 

Questions 1 and 2 of GREENHOUSE (Science example 2) are knowledge about science items. Both of these 
items belong to the category "Scientific explanations." In Question 1 , students must interpret evidence presented 
in two graphs and argue that the graphs together support an explanation that an increase in the Earth's average 
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temperature is due to an increase in carbon dioxide emissions. Question 2 asks students to use evidence from 
the same graphs to support a different conclusion. 

Attitudes towards science 

Individuals' attitudes play a significant role in their interest and response to science and technology in general 
and to issues that affect them in particular. One goal of science education is for students to develop attitudes 
that make them likely to attend to scientific issues and subsequently to acquire and apply scientific and 
technological knowledge for personal, social, and global benefit. 

PISA's attention to attitudes towards science is based on the belief that a person's scientific literacy includes certain 
attitudes, beliefs, motivational orientations, sense of self-efficacy, values, and ultimate actions. This is supported 
by and builds upon Klopfer's (1976) structure for the affective domain in science education, as well as reviews 
of attitudinal research (for example, Gardner, 1975, 1984; Gauld & Hukins, 1980; Blosser, 1984; Laforgia, 1988; 
Osborne, Simon & Collins, 2003; Schibeci, 1984) and research into students' attitudes towards the environment 
(for example, Bogner & Wiseman, 1999; Eagles & Demare, 1999; Weaver, 2002; Rickinson, 2001). 

In PISA 2006, when science was the major domain assessed, an assessment of students' attitudes and values 
was included using the student questionnaire and through contextualised questions posed immediately after the 
test questions in many units (OECD, 2006). These contextualised questions were related to the issues addressed 
in the test questions. However, since science constitutes a minor part of the assessment in PISA 2009, the 
assessment will not contain any contextualised (embedded) attitudinal items. 



ASSESSING SCIENCE IN PISA 

Test characteristics 

In accordance with the PISA definition oi scientific literacy, test questions (items) require the use of the scientific 
competencies (see Figure 3.3) within a context (see Figure 3.2). This involves the application of scientific 
knowledge (see Figures 3.4 and 3.5). 

Figure 3.6 is a variation of Figure 3.1 that presents the basic components of the PISA framework for science 
assessment in a way that can be used to relate the framework with the structure and the content of assessment 
units. Figure 3.6 may be used both synthetically as a tool to plan assessment exercises, and analytically as a tool 
to study the results of standard assessment exercises. As a starting point to construct assessment units, we could 
consider the contexts that would serve as stimulus material, the scientific competencies required to respond to 
the questions or issues, or the scientific knowledge central to the exercise. 



Figure 5.6 ■ A tool for constructing and analysing assessment units and items 
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A test unit is comprised of a group of independently scored questions (items) of various types, accompanied 
by stimulus material that establishes the context for the items. Many different types of stimulus are used, 
often in combination, to establish the context, including passages of text, photographs, tables, graphs, and 
diagrams, often in combination. The range of stimulus material is illustrated by the three units included in 
this chapter. GREENHOUSE (Science example 2) has an extensive stimulus comprised of half a page of text 
and two graphs, whereas the stimulus of PHYSICAL EXERCISE (Science example 3) is atypical in its brevity 
and reliance on visual suggestion. 

The reason PISA employs this unit structure is to facilitate the use of contexts that are as relevant as possible, 
and that reflect the complexity of real situations, while making efficient use of testing time. Using situations 
about which several questions can be posed, rather than asking separate questions about a larger number of 
different situations, reduces the overall time required for a student to become familiar with the material relating 
to each question. However, the need to make each scored point independent of others within a unit needs to be 
taken into account. It is also necessary to recognise that, because this approach reduces the number of different 
assessment contexts, it is important to ensure that there is an adequate range of contexts so that bias due to the 
choice of contexts is minimised. 

PISA 2009 science test units incorporate up to four cognitive items that assess students' scientific competencies. 
Each item involves the predominant use of the skills involved in one of the scientific competencies, and primarily 
requires knowledge of science or knowledge about science. In most cases, more than one competency and 
more than one knowledge category are assessed (by different items) in this way within a unit. 

Four types of items are used to assess the competencies and scientific knowledge identified in the framework: simple 
multiple-choice items, closed constructed-response items, complex multiple-choice items, and open constructed- 
response items. About one-third of the items are simple multiple-choice items, like Question 2 of ACID RAIN 
(Science example 1 ), which require the selection of a single response from four options. Another third of the items 
either require closed constructed-responses, or are complex multiple-choice items. Questions 1 and 2 of PHYSICAL 
EXERCISE (Science example 3), which require students to respond to a series of related "Yes/No" questions, are 
typical complex multiple-choice items. The remaining third of the items are open constructed-response items, like 
the remaining questions in ACID RAIN and PHYSICAL EXERCISE and the three items in GREENHOUSE (Science 
example 2). These require a relatively extended written or drawn response from students. 

While the majority of items are dichotomously scored (that is, responses are either given all credit or no credit), 
some of the open constructed-response items allow for partial credit, and give students credit for having a 
partially correct answer. The categories "Full credit", "Partial credit" and "No credit" divide students' responses 
into three groups in terms of the extent to which the students demonstrate the ability to answer a question. A 
Full credit response requires a student to show a level of understanding of the topic that is appropriate for a 
scientifically literate 1 5-year-old student. Less sophisticated or correct responses may qualify for "Partial credit", 
with completely incorrect, irrelevant or missing responses being assigned No credit . Question 3 of ACID 
RAIN (Science example 1) and Question 2 of GREENHOUSE (Science example 2) are partial credit items. 

The need for students to possess a degree of reading literacy in order to understand and answer written 
questions on science raises an issue of the level of reading literacy required. Stimulus material and questions 
use language that is as clear, simple and brief as possible, while still conveying the appropriate meaning. The 
number of concepts introduced per paragraph is limited and questions that require too high a level of reading, 
or mathematics, are avoided. 

Science assessment structure 

Each PISA assessment must include an appropriate balance of items assessing scientific knowledge and 
competencies. Figure 3.7 shows the distribution of score points among the knowledge of science and knowledge 
about science categories, expressed as percentages of the total score points, for both PISA 2006 (when science 
was the major domain) and PISA 2009. 
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Figure 3.7 ■ Distribution of score points for knowledge 



Knowledge of science 

Physical systems 
Living systems 
Earth and space systems 
Technological systems 
Subtotal 

Knowledge about science 

Scientific enquiry 
Scientific explanations 
Subtotal 
Total 



Percentage of score points 

PISA 2006 PISA 2009 

17 13 

20 16 

10 12 

8 9 

SS SO 



23 

22 

4S 

100 



23 

27 

SO 

100 



The corresponding distributions for the scientific competencies are given in Figure 3.8. 



Figure 3.8 ■ Distribution of score 


points for scientific competencies 


Percentage of score points 


Scientific competencies 


2006 


2009 


Identifying scientific issues 


22 


23 


Explaining phenomena scientifically 


46 


41 


Using scientific evidence 


32 


37 


Total 


100 


100 


Item contexts are spread across personal, social and global settings approximately 
is a roughly even selection of areas of application as listed in Figure 3.2. 


in the ratio 1 ;2:1 , and there 


The distributions for item types are given in Figure 3.9 






Figure 3.9 ■ Distribution of score 


points for item types 




Percentage of score points 


Item types 


2006 


2009 


Simple multiple-choice 


35 


32 


Complex multiple-choice 


27 


34 


Closed constructed-response 


4 


2 


Open constructed-response 


34 


32 


Total 


100 


100 
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Figure 5.10 ■ Summary descriptions of the six proficiency levels in science 





Lower 

score 

limit 


Examples of items 
at each level 


What students can typically do at each level 


Level 6 


707.9 


ACID RAIN Q3 
Full credit (71 7) 

GREENHOUSE Q3 (709) 


At Level 6, students can consistently identify, explain and 
apply scientific knowledge and knowledge about science in 
a variety of complex life situations. They can link different 
information sources and explanations and use evidence from 
those sources to justify decisions. They clearly and consistently 
demonstrate advanced scientific thinking and reasoning, and 
they use their scientific understanding in support of solutions 
to unfamiliar scientific and technological situations. Students 
at this level can use scientific knowledge and develop 
arguments in support of recommendations and decisions that 
centre on personal, social or global situations. 


Level 5 


633.3 


GREENHOUSE Q2 
Full credit (659) 


At Level 5, students can identify the scientific components of 
many complex life situations, apply both scientific concepts 
and knowledge about science to these situations, and can 
compare, select and evaluate appropriate scientific evidence 
for responding to life situations. Students at this level can use 
well-developed inquiry abilities, link knowledge appropriately 
and bring critical insights to situations. They can construct 
explanations based on evidence and arguments based on their 
critical analysis. 


Level 4 


558.7 


PHYSICAL EXERCISE Q3 (583) 

GREENHOUSE Q2 
Partial credit (568) 


At Level 4, students can work effectively with situations and 
issues that may involve explicit phenomena requiring them to 
make inferences about the role of science or technology. They 
can select and integrate explanations from different disciplines 
of science or technology and link those explanations directly 
to aspects of life situations. Students at this level can reflect 
on their actions and they can communicate decisions using 
scientific knowledge and evidence. 


Level 3 


484.1 


PHYSICAL EXERCISE Q1 (545) 

GREENHOUSE Q1 (529) 

ACID RAIN Q3 
Partial credit (513) 

ACID RAIN Q1 (506) 


At Level 3, students can identify clearly described scientific 
issues in a range of contexts. They can select facts and 
knowledge to explain phenomena and apply simple models 
or inquiry strategies. Students at this level can interpret and 
use scientific concepts from different disciplines and can 
apply them directly. They can develop short statements using 
facts and make decisions based on scientific knowledge. 


Level 2 


409.5 


ACID RAIN Q2 (460) 


At Level 2, students have adequate scientific knowledge to 
provide possible explanations in familiar contexts or draw 
conclusions based on simple investigations. They are capable 
of direct reasoning and making literal interpretations of the 
results of scientific inquiry or technological problem solving. 


Level 1 


334.9 


PHYSICAL EXERCISE Q2 (386) 


At Level 1, students have such a limited scientific knowledge 
that it can only be applied to a few, familiar situations. They 
can present scientific explanations that are obvious and follow 
explicitly from given evidence. 
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REPORTING PROFICIENCY IN SCIENCE 

PISA results are reported on a scale constructed using a generalised form of the Rasch model as described by 
Adams, Wilson and Wang (1997). For each domain (reading, mathematics and science), a scale is constructed 
with a mean score of 500 and standard deviation of 1 00 among OECD countries; accordingly, about two-thirds 
of students across OECD countries score between 400 and 600 points. 

When science was the major assessment domain for the first time in 2006, six proficiency levels were defined 
on the science scale. These same proficiency levels will be used in the reporting of science results for PISA 
2009. Proficiency at each of the six levels can be understood in relation to the kinds of scientific competencies 
that a student needs to attain at each level. Eigure 3.10 presents a description of the scientific knowledge 
and skills which students possess at the various proficiency levels, with Level 6 being the highest level of 
proficiency. It also gives the level and scale score of each item belonging to the three units from the PISA 2006 
assessment, which are used as examples throughout this chapter. 

e capacity of students who performed below Level 1 in PISA 2006 (about 5.2% of students on average across 
OECD countries) could not be reliably described because not enough science items were located in this region 
of the scale. Level 2 was established as the baseline level oi scientific literacy, defining the level of achievement 
on the PISA scale at which students begin to demonstrate the scientific knowledge and skills that will enable 
them to participate actively in life situations related to science and technology. 

Factors that determine difficulty of items assessing science achievement include: 

■ The general complexity of the context. 

■ The level of familiarity of the scientific ideas, processes and terminology involved. 

■ The length of the train of logic required to respond to a question - that is, the number of steps needed to arrive 
at an adequate response and the level of dependence of each step on the previous one. 

■ The degree to which abstract scientific ideas or concepts are required in forming a response. 

■ The level of reasoning, insight and generalisation involved in forming judgements, conclusions and explanations. 

Question 3 of GREENHOUSE (Science example 2) is an example of a difficult item, located at Level 6 on the 
PISA science scale. This question combines aspects of the two competencies, identifying scientific issues and 
explaining phenomena scientifically. As a first step to solving this problem, the student must be able to identify 
the change and measured variables and have sufficient understanding of the methods of investigation to recognise 
the influence of other factors. In addition, the student needs to recognise the scenario and identify its major 
components. This involves identifying a number of abstract concepts and their relationships in order to determine 
what "other" factors might affect the relationship between Earth's temperature and the amount of carbon dioxide 
emissions in the atmosphere. Thus, in order to respond correctly, a student must understand the need to control 
factors outside the changed and measured variables and must possess sufficient knowledge of "Earth systems" to 
identify at least one of the factors that should be controlled. Sufficient knowledge of "Earth systems" is considered 
the critical scientific skill involved, so this question is categorised as explaining phenomena scientifically. 

Question 1 of PHYSICAL EXERCISE (Science example 3) is an example of an easy item, located at Level 1 on 
the PISA science scale below the baseline of scientific literacy. To gain credit, a student must correctly recall 
knowledge about the operation of muscles and formation of fat in the body, particularly the facts that when 
muscles are exercised they get an increased flow of blood and fats are not formed. This knowledge enables the 
student to accept the first statement of this complex multiple-choice question and reject the second one. In this 
item, no context needs to be analysed - the knowledge required has widespread currency and no relationships 
need investigating or establishing. 

PISA 2006 results were also reported on three subscales corresponding to the three scientific competencies. 
These subscales used the same six proficiency levels as the combined scale, but with descriptors unique to each 
scale. In addition, country performance was compared on the bases of knowledge about science and the three 
main knowledge of science categories (Physical systems. Living systems, and Earth and space systems). 

While the analyses drawn from these kinds of comparisons could be valuable, caution should be used when 
relating performance to competencies and knowledge because the data come from classifying the same items in 
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two ways that are not independent. All items classified as assessing the identifying scientific issues competency 
are knowledge about science items, and all explaining phenomena scientifically items are knowledge of 
science items (OECD, 2009, p. 44). 



CONCLUSION 

The PISA definition of scientific literacy originates in the consideration of what 15-year-old students should 
know, value and be able to do as preparation for life in modern society. Central to the definition, and the 
science assessment, are the competencies that are characteristic of science and scientific enquiry: identifying 
scientific issues, explaining phenomena scientifically, and using scientific evidence. The ability of students to 
perform these competencies depends on their scientific knowledge, both knowledge of the natural world (i.e. 
knowledge of, chemistry, biology. Earth and space sciences, and technology) and knowledge about science 
itself {i.e. knowledge about scientific enquiry and scientific explanations), and their attitudes towards science- 
related issues. 

This framework describes and illustrates the scientific competencies, knowledge and attitudes involved in the 
PISA definition of scientific literacy (see Figure 3.1 1), and outlines the format and structure of the PISA 2009 
science assessment. 

Figure 5.11 ■ Major components of the PISA definition of scientific literacy 



■ Identifying scientific issues 

■ Explaining scientific phenomena 

■ U.sing scientific evidence 



■ Knowledge oj science: 
Physical systems 
Living systems 

Earth and space systems 
Technology systems 

■ Knowledge about science: 
Scientijic enquiry 
Scient^ic explanations 



Interest in science 

Support for scientific enquiry 

Responsibility towards 
resources and environment 



PISA science test items are grouped into units with each unit beginning with stimulus material that establishes 
the context for its items. The focus is on situations in which applications of science have particular value in 
improving the quality of life of individuals and communities. A combination of multiple-choice and constructed- 
response item types is used and some items involve partial credit scoring. Unlike PISA 2006, attitudinal items 
are not included in units in PISA 2009. 

PISA 2009 science results will be reported on a single science scale having a mean of 500 and a standard 
deviation of 1 00, using the six levels of proficiency defined when science was the major assessment domain for 
the first time in 2006. Level 6 is the highest level of proficiency and Level 2 has been established as the baseline 
level of scientific literacy. Students achieving below Level 2 do not demonstrate the scientific knowledge and 
skills that will enable them to participate actively in life situations related to science and technology. 
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Notes 



1 . Throughout this framework, natural world includes the changes made by human activity, including the material world designed 
and shaped by technologies. 

2. Knowledge of the design or internal working of technology artefacts (e.g. aeroplanes, engines, computers) will not be assumed. 
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This chapter describes the framework that led the design of the 
PISA 2009 questionnaires; the aim was to gather policy-relevant 
background information linked to student achievement from 
school principals, students and parents. 

The chapter presents the types and purposes of the information 
collected at four different levels: the educational system as a whole; 
the school level; the instructional setting; and the student level. It 
also puts forward ideas for analysing the policy-relevance of the data 
collected, such as: investigating effective learning environments in 
reading; ensuring school effectiveness and management; promoting 
educational equity and cost effectiveness; and developing system- 
level indicators. 
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INTRODUCTION 

Although PISA is probably best known for measuring and reporting the achievement of 1 5-year-olds in reading, 
mathematics, and science in a wide range of countries, it also serves a larger purpose for interpreting test results 
and assessing educational policies through its collection of background information. This chapter presents and 
discusses the purpose and usefulness of the Questionnaire Framework, which is used to gather the background 
information addressing policy issues linked to student achievement. 



Clearly, interest in the academic performance of students is not limited to descriptions of performance levels 
and differences within and among countries. Reading the achievement results leads to questions such as: 
Why do 15-year-olds in some countries perform better than those in others? Why are some students high 
performers, many others average, and some low performers? Can these differences be explained by differences 
in societal characteristics, families, school resources, instructional practices, and communities? Being policy 
driven, not only must PISA provide policy makers with descriptions of the results, but also with the information 
and capacity to improve them. The test results alone cannot provide answers to these questions. Therefore, 
PISA's goal is to gather additional data that will enable policy makers to draw inferences about the patterns of 
achievement results and why they occur. 



How can the additional data from questionnaires be used to interpret achievement differences? Their most 
important contribution is that they enrich the analysis by describing the characteristics of families and the 
organisation of education systems at different levels from the national level through to the classroom and home. 
PISA sets out to relate educational achievement to these characteristics in order to identify general patterns 
among countries and specific patterns within countries. Given that the PISA 2009 results can be compared 
with those of the earlier studies, (PISA 2000, PISA 2003 and PISA 2006) PISA can not only compare the trends 
in achievement over almost a decade for many countries, but also attempt to relate those trends to changes in 
policies, practices, and student populations. 



PISA provides the possibility to speculate about the change in achievement outcomes. PISA is speculative 
because it is what is known as a yield study that attempts to ascertain the cumulative achievement of students 
towards the end of compulsory education. In other words, PISA is not an attempt to ascertain how much 
learning has taken place in the secondary school in which a student is enrolled. This would require comparing 
the achievement of students at entry into their present school to their achievement at that same school at 
age 1 5. This would provide a measure of progress or value-added in achievement associated with educational 
experiences in that particular school. PISA only tests the students once at age 15. A further challenge is the 
relatively short period of time that students in many countries have attended their present secondary schools, 
typically three years or less for most students. Therefore, only a portion of their level of achievement is due to 
educational characteristics of their present school and classroom, while some is due to previous educational 
experiences that may not be reflected in data from the present school. 



In essence, PISA makes it possible to find patterns of statistical association between achievement, on the one 
hand, and family, school, and other educational influences, on the other. Some of these patterns will be strong 
and relatively consistent among countries, and those are most likely to tell an underlying story that may initiate a 
search for causal mechanisms. Others will be weaker and inconsistent from country to country and may or may 
not have important policy implications. Interpretation for policy purposes must be sensitive to these differences. 
When the patterns of association are strong, there is also a basis for further investigation by countries to confirm 
whether such a phenomenon exists when more direct research is applied. Such focused research can involve 
intensive qualitative studies of families, schools, and classrooms, or it can employ statistical models to measure 
achievement gains through value-added studies or even experimental investigations that focus on associations 
found in PISA. 



Educational policy is complex and is not easily encapsulated by a single definition. In general, educational 
policy addresses the actions that families, government bodies, and other educational organisations take 
to shape and operate educational systems at all levels of education and all levels of government. Clearly, 
educational policy embraces the actions of many entities including those who occupy positions of educational 
leadership such as ministers and secretaries of education, those who make laws, technical staff who make 
operative and concrete decisions, and administrators and teachers who must implement specific educational 
actions. In addition, educational policy may include the activities of families through mandates or guidelines 
that influence their behaviour. 



150 
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In terms of some of the general uses of the results, policy makers are using PISA findings to gauge the 
knowledge and skills of students in their own country in comparison with those of the other participating 
countries; establish benchmarks for education improvement, for example, in terms of the mean scores 
achieved by other countries or their capacity to provide high levels of equity in educational outcomes and 
opportunities; and understand relative strengths and weaknesses of their education systems. (OECD, 2007) 

Good educational policy is informed educational policy in which all of the responsible actors (policy makers, 
school principals, teachers, students and parents), are provided with the knowledge that they need to make good 
educational decisions. Although the test results from PISA may inform and motivate these actors to seek ways of 
improving the levels and equality of educational performance, they need considerable further information that 
will assist them in formulating strategies to achieve those ends. 

For this reason PISA has made a concerted effort to gather background information on educational systems, 
schools, families, and students that might inform the potential sources of differences in achievement, both 
within and among countries, and that might be used to formulate strategies for improvement of overall academic 
performance and educational equity. 

Choosing and collecting the types of specific background information on families, students, and schools that 
might be linked to achievement is a daunting task. The reason is that there are many possible types of information 
that can be sought, especially those which may be pertinent in some countries and regional settings, but not 
in others. Further, there is the constraint on time of those who are asked to provide the information, where 
most of the specific information on schools and classrooms must be obtained from individual students and 
principals through survey questionnaires. In order to safeguard the time requested of both groups of informants 
and to increase the reliability of the information and the completeness of responses, it is important to limit the 
typical time required of respondents to provide questionnaire information. Principals are usually very busy, 
so it is essential to ask them for information on key questions that help to understand school policies and 
operations. Students will not only be asked to fill out background information on their families and school 
experiences, but will also take PISA tests in the subject areas - a taxing demand on time and effort. To gather 
background information, only about 30 minutes can be allotted to the questions asked of students on their 
personal background, their learning habits, characteristics of their school and classrooms, and their attitudes to 
learning as well as their engagement and motivation. This is in addition to two hours devoted to the testing in 
the three subject areas. 

It is well known that as the length and difficulty of a questionnaire increase, the accuracy of responses tends to 
decrease, and it is increasingly likely that some questions will be skipped completely. Accordingly, the goal of 
designing surveys to obtain useful interpretive information is to prioritise the information sought and to obtain it 
accurately, subject to the time constraint for questionnaires set by the PISA Governing Board. The section below 
describes the strategy for reaching this goal. 



TYPES OF BACKGROUND INFORMATION AND THEIR PURPOSES 

The types of background information collected on the questionnaires are purpose-driven in that they aim to 
provide interpretive frameworks and data for analysing PISA results. The challenge of collecting such information 
is complicated by the fact that educational systems have many actors at different levels. For most systems there 
is a national or federal entity that sets country policy and is instrumental in allocating resources. Many countries 
also sponsor regional agencies that are partially accountable for these functions. Some schools are embedded in 
districts that monitor and enforce policies as well as distribute and contribute to school resources. It is possible to 
consider this level as the educational system as a whole, which describes the overall national or federal system 
and its supporting and regulatory elements for governing and providing education. In some countries the system 
is highly centralised under a ministry with ultimate authority, and in other countries the national responsibility for 
education is decentralised through shared responsibilities at different levels of government. However the overall 
system is organised, it ultimately must interact and support the key learning organisational site: the school. 

It is at the school level where the educational system interacts directly with communities, parents, and children. At 
school the students receive their formal educational experience. Individual schools are responsible for educating all 
children efficiently and meeting the goals and educational requirements set by higher levels of governance. They 
must recruit personnel and acquire the materials that will enable the educational process to proceed. In addition, they 
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must respond to national and regional systems of accountability for educational results in terms of accommodating 
student enrolments and producing achievement and ultimate educational attainments. They encompass what is 
known as the educational core, where the teaching and learning takes place, and they must be organised to do 
that effectively. School policies are central to this mission in terms of the time that is devoted to learning, including 
the length of the school day and school year, the organisation of the curriculum, teacher selection, retention and 
professional development, parental involvement, school leadership, and the internal allocation of resources. 

Schools are organised into subunits of grade levels and classrooms. Although the overall school environment 
and practices clearly have an impact on a child's education, the most intensive educational experiences take 
place within his or her grade level in classrooms. Within the school, learning objectives and practices must be 
articulated among grade levels so that the requirements for success at higher grades are satisfied by learning 
at the preceding grade levels. Regardless of the educational requirements set by nations, regions, and schools, 
these must be translated into specific practices which often differ in significant ways in application from teacher 
to teacher and classroom to classroom. In recent decades educators have recognised that it is not only the 
guiding policies which determine the quality and depth of instruction and its focus, but also the implementation 
of those principles. Research studies have found remarkable variance both among and within schools that are 
governed under the same regulations and overall decisions on practice (Fullan, 2007). It is therefore important 
to understand the classroom practices employed by teachers in such instructional settings, as well as their 
beliefs, attitudes, skills, and training experiences to establish links with student achievement. 

The first round of the new OECD Teaching and Learning International Survey (TALIS; OECD, 2009), focused 
on policies and practices that shape the teaching and learning environment in schools, providing policy- 
relevant insights on teachers' professional development, appraisal and feedback to teachers, teaching beliefs 
and classroom practices and school leadership. In future cycles of PISA and TALIS, a potential linkage of the 
two programmes on the school level could lead to analytical gains in a country. First, the student background 
and student performance information from PISA would provide an important context for analysing and better 
understanding the teacher responses in TALIS (e.g. understanding the extent to which teachers' professional 
development needs being met vary between high and low performing schools or schools facing more or less 
challenging circumstances). Second, teacher information, such as that collected through TALIS, could improve 
the measurement of the instructional environment in schools and enhance the explanatory power within PISA 
both in terms of school performance and the attitudes of students. 

Finally, the level which is perhaps the most complex of all is the student level. National and regional entities 
set goals, institutional arrangements, financing, and licensing requirements for educational personnel. Schools 
operate within these parameters to create learning organisations. Most implementation of learning takes place 
within instructional units within the school, typically classrooms. But, learning takes place because all of these 
arrangements create opportunities, incentives, and expectations for individual students and student groups to learn. 
What is reflected in the achievement measures is the product of how these specific provisions operate to enable 
students to succeed. Therefore at the individual student level, it is important to document the student backgrounds, 
student expectations and attitudes, available resources and practices that affect educational outcomes. 

Knowing a student's educational experiences in the school or classroom is not adequate for understanding 
all student influences on learning. Fifteen-year-old students have spent the vast majority of their waking 
hours outside of school. This means that differences in home and community environments such as parental 
educational resources in the home and other learning opportunities are crucial determinants of student learning 
and achievement. Furthermore, family influences are likely to affect student motivations and attitudes towards 
learning overall and in specific subjects and can have profound effects on performance, particularly in reading. 
Since most out-of-school learning depends on parental and community resources, such background information 
is crucial in interpreting achievement patterns as well as students' capacity and attitudes towards learning. 

What is described here is a constellation of institutions and actors at different levels that constitute an 
educational system. The interactions among these are complex and not fully understood, yet each level is 
important in shaping educational policy and probable education outcomes. The principal challenge of obtaining 
useful background information to help interpret achievement results and relate these to educational policy is 
determining the types and form of information to be collected and how to relate it to student achievement in 
what appears to be a swamp of complexity. To add to the challenge, most of the information must be collected 
directly from principals and students in a highly time-constrained setting where their time is precious and for 
which only very limited demands can be made. 
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The strategy is to begin with some of the major relevant questions at each level of the educational system and to 
use those questions to focus on important data categories that are central to the later analysis of achievement. 
These must be translated into specific questions that appear on the questionnaires, or for which answers are 
available from other sources, and that the respondents understand so that accurate answers will be provided 
within time constraints. Since this process begins with some overall policy questions and some knowledge of the 
general literature that establishes relationships between this information and student achievement, it is important 
to establish the required types of information at each of the four levels of the system. The four levels are: 

■ educational system as a whole 

■ school level 

■ instructional settings 

■ individual student level 

Educational system as a whole 

The purpose of information at the highest level is to provide comparative descriptions of educational systems 
and to allow for the exploration of a large range of educational outcomes as they relate to the overall conditions 
that characterise the educational system. Examples of descriptive information include: 

■ measures of wealth and income of the country and region 

■ overall status of teachers 

■ community involvement in schooling 

■ level of decision-making 

■ public versus private schools 

■ measures of societal inequality 

■ forms of accountability 

What is unique about this list of information is that much of it can be obtained from sources other than questionnaires 
because it is available as administrative data from the countries themselves as well as international organisations 
such as the OECD, UNESCO and the World Bank (OECD, 2008; UNESCO, 2005; World Bank, 2008). 

Each category of information is related to social science studies that show a relation between student achievement 
and the information category. For example, studies of educational achievement among societies find that both 
within and among nations the amount of per-capita income is a strong predictor of student achievement (Baker, 
Goesling & Letendre, 2002). The overall status of teachers is important because it affects the attractiveness of 
the teaching profession (Darling-Hammond, 1997). Active community and parental involvement in schooling 
has been linked to student educational outcomes (Ho & Willms, 1996). Measures of societal inequality are 
linked to educational inequality in terms of family resources and learning conditions (Rothstein, 2004). And, 
the measurement, visibility, and sanctioning of student and school success are believed to provide inducements 
to better performance (Hanushek & Raymond, 2004). 

These are antecedents of educational policy, operations, and outcomes in the sense that they exist in a larger societal 
context which may affect educational results. For example, measures such as Gross Domestic Product (GDP) per 
capita reflect family resources that influence education as well as societal investments in education. Heterogeneity 
of the population through immigrants, and particularly immigrants from other language, cultural, and economic 
backgrounds, will determine the educational challenges in classrooms as well as the preparedness of students to 
succeed in a mainstream environment that may differ significantly from thatof their population subgroups. Community 
involvement in schooling can provide greater effectiveness through educational adaptation to the needs of local 
populations. Throughout the world many countries are establishing different approaches to decentralising decision- 
making, differences that might have a bearing on educational achievement in terms of the level of the educational 
system where such matters as curriculum, teacher salaries, and student policies are established. Although the 
centralisation versus decentralisation of decision-making will be addressed more fully when its impact at the school 
level is discussed, it is important to note that the authority to make decisions at each level is usually established by 
national, federal or regional governments, and schools are expected to follow these laws, regulations, and conventions. 
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Another form of decentralisation is the degree to which non-government schools are relied upon to provide 
^ education. The use of a market of choice among non-governmental (private or independent schools) is an 
^ important form of decentralisation of decision-making. Although such schools exist within a government 
regulatory framework, they generally have considerably more freedom than public schools to make educational 
decisions. Indeed, in some countries the private schools are fully funded from government sources (government- 
dependent private schools) and must meet certain government standards Games, 1984). Since families can 
choose among such schools, they also express their preferences, a particularly strong form of involvement 
in educational decisions. Choice among public and private schools (whether independent or government- 
dependent) is championed as a mechanism to create incentives for schools to compete for students and improve 
the overall performance of the educational system (UNESCO, 2005). By providing background information on 
the organisation of schools, it is hoped to gain insights into the impact of alternative structural arrangements 
on achievement. Accountability systems are another focus of educational systems in most countries where the 
attempt is to put in place systems of school and student assessment that will ensure that students meet important 
educational standards (Carnoy, Elmore & Siskin, 2003). In some countries the passing of examinations at the 
end of courses or end of the school year are required to move to the next level of study. In other cases, tests 
are given to establish how well individual students and schools are learning, but the results are simply used to 
provide evaluation information without individual consequences. Examination policies also differ with respect 
to whether the tests are developed by classroom teachers and schools for their own students or by regional, 
federal or national authorities. That is, some examinations are internal to the school and focus on classroom 
or school objectives, and others are external where the national, federal or regional authorities set a standard. 
Different forms of examination may have different consequences for providing incentives to learn, uniformity of 
what is taught, and educational results. 

Specific patterns of educational investment among regions, schools, and students may have important 
consequences as well. In some countries policy provides for similar funding per student for all students, regardless 
of the economic circumstances of their families. In others there are additional resources for students from rural, 
impoverished or immigrant backgrounds or for those with learning or physical difficulties. How spending is 
allocated may have implications for both the level of achievement as well as equity in achievement outcomes. 

Overall, information on the educational system as a whole can provide a picture of how the overall resources 
and practices of a society impact educational achievement. For example, to what degree do differences in the 
overall wealth of a society and its distribution affect the level and distribution of achievement? Do the impacts 
differ by subject and population group such as natives and immigrants? To what degree does the schooling 
system serve to equalise educational outcomes, and to what degree do the outcomes simply reflect existing 
inequalities. How attractive is the teaching profession relative to other professions in the country, and how 
does this affect the ability to maintain a quality teaching force? How does the organisation of schools, school 
decision-making and investment policies among different groups of students and regions affect educational 
results? All of these are questions that can only be addressed if background information is available that 
supplements the data on achievement. 
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Tgble 4.1 Examples of data items for educational system as a whole 



Title 


Description/measure 


Measures of wealth and income of 
country and region 


Cross domestic product per capita 


Overall status of teachers 


Teacher salaries and benefits relative to other 
occupations with similar education 


Community involvement in schooling 


Extent of influence of parents and school governing 
board on decision-making 


Level of decision-making 


Government levels or constituencies that influence 
directly decisions in staffing, budgeting, instructional 
content, and assessment practices 


Public versus private schools 


Is school public or private? 


Measures of social inequality 


Summary measures of income distribution 


Forms of accountability 


Are achievement data used to monitor accountability? 



(OECD, 2008; UNESCO, 2005; The World Bank, 2008) 



School level 

As emphasised earlier, the organisation of schools is complex and varies considerably, not only from country to 
country, but also among internal educational sub-divisions such as federal states or provinces, regions, school 
districts, and individual schools. PISA aims to identify the types of differences that are most likely to impact student 
achievement and selects the differences to be measured in a limited time frame for questionnaire administration. 

Since PISA is concerned with relating school characteristics to their probable efficacy in improving student 
achievement, in most cases measures of actual school operations are sought rather than the formal policies 
under which they are supposed to operate. In recent decades it has become widely recognised that the formal 
descriptions of educational operations in schools are often misleading because implementation differs so 
widely (Fullan & Stiegelbauer, 1 991 ; Fullan, 1 992). Implementation refers to the actual application of schooling 
procedures, and researchers have found that under the same descriptive rubrics the actual behaviour can differ 
markedly. This means that official lists of educational goals, requirements, and procedures are not adequate to 
describe the operational behaviour of schools and must be supplemented by information from the respondents 
on the actual ways in which their schools operate. Accordingly, the questionnaire surveys attempt to obtain 
information directly on operations from the participants, and especially the school principal. 

Within this context, some of the main categories in which PISA 2009 attempts to gather background information 
on schools are; 

■ school leadership 

■ composition of student enrolments 

■ curricular emphasis 

■ extra-curricular activities 

■ school size 

■ support for teaching and learning 

School leadership is a particularly important variable that might be altered by policy in directions that support 
learning. Some success has been found in research on this issue where attempts have been made to measure 
principals' activities in key dimensions of a school's instructional programme and to relate these to student 
outcomes such as reading achievement (Hallinger, Bickman & Davis, 1996). The goal is to obtain information 
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on principals who are devoted to and active in attending to those dimensions that seem effective in improving 
student learning. 

Furthermore, it is worthwhile to ascertain whether the important dimensions of school leadership that affect 
achievement are relatively universal in nature or differ according to educational system and type of student. 
This is relatively new research in the educational policy area, but establishing an international database is the 
first step in using it. 



A related component of school leadership is imposed on the school by the degree of decentralisation of 
decision-making set out for schools. In recent years there has been considerable focus on school autonomy 
as a way of overcoming bureaucratic rigidity and potentially having an impact on learning outcomes 
(Bottani & Favre, 2001 ; Chubb & Moe, 1 990). In some societies the hiring of teachers, establishing curriculum, 
and assessing students is carried out at a central level where the school is merely expected to follow the 
directions set out for school routine operations. At the opposite end of the spectrum the school is largely 
autonomous and expected to choose its own teachers, to influence teacher compensation, to choose curriculum 
and undertake its own academic assessment of students. Of course, when school is referred to in this sense, 
it is not only referring to the head administrators at school sites, but also to the roles that teachers, parents, and 
students might play in educational decision-making. Whether decisions should be centralised or decentralised 
and which ones should be placed at different levels is still a subject for debate. However, by getting measures 
of centralisation and decentralisation for different functions, it may be possible to ascertain the impact of such 
policies on achievement. 



The type of student body is likely to have an impact on educational achievement. Not only does the individual 
socio-economic status of students have an impact on the performance of individual students, but also the overall 
socio-economic composition of school enrolments. Schools with students of higher socio-economic status tend 
to create an overall school environment of high aspirations and support for academic learning which provides 
advantages for all students, regardless of high or low socio-economic status, who are enrolled in those schools. 
This is known as a compositional or peer effect. But, even beyond the matter of how student composition affects 
individual students, such schools may attract better teachers, those who seek school environments where they 
will experience success in student learning. Furthermore, teachers in schools with higher socio-economic 
enrolments know that the overall academic level of performance is higher and are likely to respond by setting 
a faster pace for the curriculum and higher performance expectations for all students. For these reasons it is 
important to assess the overall socio-economic composition of students to explore its impact on achievement 
(Zimmer &Toma, 2000). 



A somewhat different dimension of the student body is the proportion of immigrants and their origins. In general, 
immigrants face a number of disadvantages such as having a mother tongue different from the official language 
of the school and the resident society. In addition, they often have to adjust to an unfamiliar cultural context and 
they may face discrimination. Immigrants who have emigrated from low income countries, where educational 
opportunities are slim and standards low, face a particularly harsh educational challenge. When a relatively 
high proportion of school enrolment is comprised of children of immigrants, the overall learning environment 
of the school may be challenged for all students by a slower curriculum pace, language deficiencies of fellow 
students, and differential educational aspirations. Therefore not only is it important to take into account the 
socio-economic composition of student enrolments, but also the concentration of immigrant students and 
especially those from countries with low income and poor educational opportunities. 

Two other conditions that may have an impact on determining the supportive nature of the school environment 
are the affluence of the surrounding community and the degree of parental involvement. Affluent communities 
tend to have more positive educational influences and support systems in comparison to poorer communities. 
First, they tend to be safer with less crime and violence. They are therefore less preoccupied than poor 
communities with the possible effects of crime and violence on students. Second, such neighbourhoods have 
more resources to support schools in informal ways through volunteering and the occasional provision of extra 
resources that may be needed. Reciprocally, good schools reflect on the quality of neighbourhoods, setting out 
incentives for neighbourhoods to assist in their maintenance and support. 

Parental involvement has an even more direct effect on student achievement (Epstein, 2001). When schools 
encourage and make a provision for parental involvement, parents can become more effective in supporting both 
school programmes and the educational progress of their own children. Furthermore, parents who participate in 
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school activities are more likely to volunteer their efforts in assisting the school, therefore increasing available 
resources. Once parents are involved in addressing the needs of a school, they may also seek other ways to 
assist the school in acquiring resources and assisting those children in academic need. Parents who know what 
schools expect through familiarity with the school programme and teacher expectations are better able to assist 
their own children in learning. In some schools, parental involvement can take the form of teaching parents the 
skills that they need to support their children's learning. 

Finally, the act of attracting parents to participate actively in educational and school endeavours can serve to 
form social networks where parents get to know and help each other. That is, parents become a community 
of resources that can assist each other to address a wider range of child and family needs, a resource that is 
particularly important for those who would otherwise be isolated and with fewer resources to draw upon. It has 
been argued that such social networks built around schools raise overall achievement through the accumulation 
of what is known as social capital (Coleman, 1988). 

Beyond what happens in the classroom, schools as a whole can also put curricular emphasis on particular subjects 
such as reading. Some schools require all teachers at each grade level to devote a certain amount of time to both 
the teaching of reading and the allocation of reading as part of assignments and school activities. Others even 
schedule independent reading time during the school day or coordinate school work with library resources and 
services. The prevalence of other literacy activities such as writing, listening, and requiring oral presentations 
also reinforces reading skills. Clearly it is expected that more time would be devoted to reading assignments and 
literacy activities to contribute to reading proficiencies. The same is true with regard to an emphasis on critical 
reading skills that require more sophisticated comprehension and understanding of written materials. When the 
curriculum approach requires critical analysis of readings through questioning a point of view, understanding 
the different sides of an argument and the evidence supporting each side, analysing the literary styles and 
meaning, and other deeper approaches to meaning in reading, the student is exposed to a richer understanding 
of written material and thought. Above all it is not only the specific amount of reading and assessment of narrow 
components of reading that constitute differences in school curriculum, but also the emphasis placed on 
preparing students for a wide array of literacy tasks such as writing, debating, problem-solving, and discussion. 
New demands are also set by media. Proficiency in reading is not limited to printed text but also to the texts in 
the electronic media, which are becoming a more important part of student life and societal communication 
in general. PISA 2009 has included an assessment of literacy in the reading of electronic texts in addition to 
printed texts (see Chapter 1). 

On a broader level, the variety and quality of extra-curricular activities can also contribute to reading proficiencies. 
Some extra-curricular opportunities depend intrinsically on reading and literacy such as drama and theatre, 
journalism, creative writing, the school newspaper and professional clubs such as those devoted to science, 
history, communication technologies or a foreign language and culture. To the degree that these require reading 
activities as part of the extra-curricular offerings, reading ability should be enhanced as a by-product. 

School size is viewed widely as having some relation to achievement, although the overall relationship is not clear. With 
larger enrolments, schools can offer their students a greater variety of courses and electives as well as extra-curricular 
activities. This means that individual students have more options when choosing courses and also, possibly, a greater 
choice of teachers for particular subjects. By increasing the possibility of electing more courses of their choice, students 
can select those that are most meaningful to them educationally and for which they are most motivated. 

But, larger schools also tend to be more impersonal where students may not feel supported as much by teachers 
and other personnel. Research has found that smaller high schools demonstrate a greater and more equitable 
distribution of student engagement and achievement among 15-year-olds (Lee & Smith, 1995). Thus, the size 
relation goes in both directions, and there is no overall consensus in the literature on where the balance 
lies. In fact, some research suggests that size affects different groups of students (e.g. socio-economic groups) 
differentially, so the best size policy depends upon the characteristics of the students (Lee & Smith, 1 997). 

Finally, a highly supportive teaching and learning environment at the school level enhances the teaching and 
learning of all subjects. Studies of effective schools find that an academic focus on school culture is positively 
related to student achievement. In such an environment it is clear what is valued by both teachers and students. 
Academic activities and student academic performance are considered to be a central feature of a successful 
school. This literature has also identified a range of other features of schools that are associated with greater 
success on academic measures (Scheerens & Bosker, 1997; Sammons, 1999; Taylor, Pressley & Pearson, 2002). 
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Tgble 4.2 Examples of data items for school level (see Annex B) 



Title 


Description 


School leadership 


Principal's activities and behaviours 


Composition of student enrolments 


Percentage of students whose first language is not the 
test language 


Curricular emphasis 


Time spent in school on specific subjects 


Extra-curricular activities 


Inventory of school activities offered to students 


School size 


Total school enrolment 


Support for teaching and learning 


Principal's activities and behaviours on teaching and 
instruction 



These school features are important dimensions of the overall learning environment that have been linked 
to student achievement generally and reading achievement specifically. Each has been refined into specific 
questions which are used in the questionnaire of the school principal. With the aim of obtaining high quality 
data on schools, it is hoped that a range of school policies can be reviewed in many countries to ascertain their 
potential links to student reading achievement in PISA 2009. But, within the school it is the instructional setting 
in which the most intensive teaching and learning activities take place. 

Instructional Settings 

There are two principal instructional settings: the home and the classroom. Obviously the home is an instructional 
setting where parents provide opportunities in the broader language area by interacting with their children, and 
in a narrower sense by promoting literacy and reading skills and reading and discussion opportunities. Often 
the availability of reading materials in the home or the use of public libraries is included as an indicator of 
support for the opportunity to read outside of school. But, governments typically have less policy leverage over 
family activities that promote reading. 

The more prominent instructional setting over which governments and educators have policy discretion is the 
school. Although students can be viewed as having received their schooling in a particular school, most of 
a student's educational experiences at school occur in the classroom. In the classroom the student has been 
exposed to subject content, curriculum materials, and instructional strategies of specific teachers. Typically, 
each teacher is specialised by subject area at the secondary level and focused on teaching courses in their 
specialised areas of the curriculum. Given that so much time is spent in the classroom and the experience is 
intense, it is likely that the strongest influences of a school on a student's experiences and achievement could be 
found in this smaller instructional setting. It is therefore important to document the characteristics of teaching 
and learning and the interactions that take place within the classroom. This section identifies some of the 
characteristics of the instructional setting that are believed to influence achievement. 

Some principal categories in which PISA 2009 has attempted to obtain information on instructional settings include: 

■ class size 

■ composition of class enrolments 

■ teacher quality 

■ opportunity to learn 

■ orderliness of classroom environment 

■ supportive teaching and learning conditions 

Although there are many dimensions of instructional settings that may affect educational outcomes, the 
most important features are usually thought of as the classroom context, the teacher, the curriculum, and 
instructional strategies. The classroom context refers to the size of classes and instructional groupings as well 
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as the characteristics of fellow students. It could also extend to the suitability and condition of the physical 
environment of the classroom. The teacher is characterised by: education and training; teaching experience; 
pedagogical and subject knowledge and skills; and attitudes towards students, subjects, and instructional 
approaches. The curriculum refers to the content of the subject that is taught as well as the supportive materials. 
Instructional approaches refer to methods of pedagogy such as the balance that teachers establish between: 
teacher lectures and whole group instruction; individual student practice and reinforcement; group activities; 
and independent projects. 

Previous research has found that the complexity of each of these areas limits the comprehensiveness and 
accuracy of information obtained by any single data collection strategy. Information by survey questionnaires is 
further reduced by the time constraints imposed by PISA to avoid excessive time demands on respondents. More 
detailed information would require long individual classroom observations as well as interpretive interviews 
with teachers and students: techniques that are costly and time-consuming and beyond the resources available. 
Nevertheless, considerable valuable information that appears to be linked to student achievement can be 
obtained through PISA's principal and student surveys. Furthermore, it is possible for individual countries to do 
more intensive studies of classrooms and schools when merited by strong findings from PISA (LeCompte, 1 999; 
Creswell, 2009). 

One of the most common measures of academic quality believed to affect student achievement is the size 
of the class. Parents prefer their children to be placed in small classes, and class size is widely used to assess 
the quality of a school. Logically, if a teacher has fewer students in the class, more time can be devoted to 
the educational needs of each student. Furthermore, fewer students mean fewer assignments to evaluate, so a 
teacher can provide more feedback and assign richer activities and assessments. Smaller classes also enable 
greater opportunities to participate in class. Teachers can call on more students within a given time period, 
and students are more likely to be called upon when they offer to respond to questions. If teachers adapt 
their instruction from teacher lectures to more participative forms of instructions with students, they might get 
further gains from smaller classes than from maintaining traditional instruction approaches used with larger 
groups of students. This illustrates the difficulties of separating out the distinct effects of class size from those of 
instructional approaches. 

An important feature of the classroom context is the composition of students in the class. At least two dimensions 
are significant, the socio-economic and immigrant composition and the levels of ability. The concern about the 
socio-economic and immigrant composition of students at the classroom level is similar to that at the school 
level in terms of its impact on learning. When a class has a greater proportion of students from more advantaged 
families and with fewer immigrants, especially those from lower-income countries where the mother tongue 
differs from the instructional language, there are positive contextual impacts on achievement that even help 
the few immigrant or lower socio-economic students in the class. The best teachers often seek these teaching 
opportunities and set high expectations and a more demanding curriculum with deeper focus and richer content 
to meet the perception of highly capable students, providing positive academic advantages for all students in the 
class. In a class with higher proportions of immigrants and students from lower socio-economic backgrounds, 
teachers adjust expectations and curriculum challenge downward, often in subtle ways. At the same time, there 
are peer interactions among students through shared ideas, vocabulary and aspirations that influence each other. 

However, one could question why the peer composition of a classroom should differ from peer composition of 
the school as a whole. This applies to schools at all levels of the socio-economic spectrum. Therefore it might 
seem that schools with a student composition that is largely lower socio-economic in nature might have similar 
student composition in its classes. But, schools follow different policies with respect to the assignment of 
students to classes. Some schools stream or track students into classes with other students of similar background 
and academic performance. In those cases, the school sorts students into classes of different academic ability 
or native language proficiency - usually a measure of achievement rather than innate ability or immigrant status. 
Although the rationale is to enable each teacher to improve instruction by tailoring it specifically to student 
ability, there is considerable evidence that the educational impact of such grouping may widen the gap in 
achievement among such groups (Oakes, 2005; Camoran, 1992). 

A central concern in explaining differences in student achievement is the quality of the teacher (Wayne & 
Youngs, 2003). The most obvious characteristics that are assumed to determine teacher effectiveness are the 
quality and depth of their education, experience, and other teaching qualifications such as knowledge of 
subject matter and effective pedagogy, particularly as they contribute to proficiency among students. However, 
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the teacher's approach to instruction and attitudes towards their students and subject are also central. There 
^ are many other dimensions of classroom activity that are under the control of the teacher and can contribute 
^ to learning such as maximising the active learning time of students, using a variety of approaches, setting high 
expectations, and providing necessary support for different types of learners. These, and related aspects of 
teacher behaviour, are obtained through the student and principal questionnaires. 

PISA 2009 also makes a special effort to ascertain the opportunity for students to learn. If they are given fewer 
opportunities to learn a subject, they are less likely to learn as much than if they were given more opportunities. 
There is therefore a focus on whether students have been exposed to different types of reading tasks and in 
different subjects. Student opportunity to improve reading depends not only on opportunities in the classroom, 
but also on those outside of school, and, especially, in the home. These types of opportunities are also likely to 
be influenced by the priority for reading development set by the school and imposed at the school level. 

An orderly classroom environment has been identified as a key condition of learning in the many studies on 
effective schools. Such an environment is relatively free of disruption, has many predictable aspects in terms of 
activities and expectations, and is focused on student achievement. 

Teaching and learning conditions with respect to reading are also central, though most of the knowledge base on this 
subject has been developed at the elementary school level. Because students are introduced to reading and reading 
development at the elementary level, the subject is often taken for granted at the secondary level as one that has 
already been accomplished. What this assumption ignores is that text at the secondary level is more complex and 
demanding with respect to achieving meaningful comprehension, and it is also more varied in its demands on the 
student from subject to subject. For example, reading skills for science, history, mathematics, and foreign languages 
may differ significantly in the structure of the material, the concepts that must be mastered, and the vocabulary. 

To further complicate matters, there is not always agreement on whether reading is even a subject at the 
secondary school level or whether it is just part of all subjects, in which case many of the subject-matter 
teachers have not been trained to diagnose or focus on reading development. Accordingly, PISA 2009 has set 
out several dimensions to evaluate specifically the teaching and learning conditions in the instructional setting. 

A different dimension of classroom learning is whether the instructional strategies take meta-cognition into 
account: the fact that individuals have different ways of learning, and pursue different subjects and tasks that 
call for different ways of learning (Baker & Brown, 1984; Flavell, 1979; Artelt, Schiefele & Schneider, 2001; 
see also Chapter 1). Meta-cognition or knowledge of one's usual or most appropriate learning strategies is not 
only helpful for students to learn efficiently, but appropriate for teachers to use in creating effective strategies 
for different groups of students according to their meta-cognitive processes. It is also believed that students can 
be oriented towards different approaches to meta-cognition that can broaden their own approaches to learning. 
When meta-cognition is taken account of, teachers and students consider the most appropriate learning 
strategies for achieving. 

Other dimensions of the classroom that should betaken into account are the challenge of the learning environment 
and the modes and media that are used in activating students to undertake reading activities. Along with these are 
issues of monitoring and feedback with regard to providing teachers and students with information on how well 
students are performing academically and how to intervene to improve learning. Such feedback must be timely 
and sufficiently frequent so that both are informed and can continue or change strategies. 
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Tgble 4.3 Examples of data items for instructional settings (see Annex B) 



Title 


Description 


Class size 


Number of students in class of test subject 


Composition of class enrolments 


Aggregation of student reports on family 
background 


Teacher quality 


Teacher's instructional activities, as described 
by students 


Opportunity to learn 


Frequency of different reading activities 


Orderliness of classroom environment 


Frequency of disruption or disorder in class 


Supportive teaching and learning conditions 


Perceptions by student of teacher interest and 
support 



Student level 

Finally, PISA obtains information at the student level to ascertain individual differences among students that can 
account for differences in educational achievement. 

Principal categories of information include: 

■ socio-economic background of students 

■ immigration status of students 

■ student learning styles 

■ student attitudes and reading activities 

It is well-known that students of different socio-economic backgrounds enter schools with different levels of 
literacy and vary in their preparedness for learning to read and acquiring a high level of reading skills. Since PISA 
students are 15 years old, these socio-economic differences have had a historical impact reflected in different 
literacy levels at the time of school entry as well as in their influence on potentially differential rates of learning 
in the years prior to sitting the PISA test. More educated parents are able to provide a richer set of oral and 
written literary experiences which contribute to the education of their children both before their children begin 
school and during school. They are also able to provide more access to written materials for reading as well 
as travel and other resources that engage their child's curiosity. By the age of 1 5 it is not only the accumulated 
effects of schooling that contribute to their children's reading proficiencies, but also the experiences that take 
place in the home. One of the main purposes of gathering student data on family background is to take into 
account these influences on learning which are not attributable to the school. 

Even so, it is often difficult to separate out the educational influence of family background from school variables 
for several reasons. Schools in different countries tend to be highly segregated by social class. That is, families 
of higher socio-economic origin typically select schools for their children that have better resources and 
educational reputation as well as a high concentration of peers with similar socio-economic backgrounds. In 
contrast, students of lower socio-economic origin are more typically found in schools with fewer educational 
resources and a high concentration of students with fewer socio-economic advantages. Since the characteristics 
of schools and those of families overlap substantially, it is difficult to separate their unique influences. And, to 
complicate matters there are often interactions between family and school resources that, in combination, have 
larger effects than when each is considered separately. 

PISA attempts to collect extensive background data on students regarding socio-economic measures such as 
parents' occupations and education, home resources, immigration status, and reading engagement outside 
of school. This information is used to adjust for differences in achievement that may be due to opportunities 
outside of the school to attempt to more closely identify school and classroom outcomes. Student attitudes and 
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activities towards learning in general and reading in particular - how they use their opportunities and interest 
in reading and how they value the importance of reading - are important interpretive measures for assessing 
differences in reading achievement. It is not always known how to establish policies to alter these attitudes 
and reading activities, but with greater knowledge of which are important, it may be possible to consider 
approaches to alter student perspectives on learning and, consequently, to raise their level of achievement. 



Tgble 4.4 Examples of data items for student level (see Annex B) 



Title 


Description 


Socio-economic background of student 


Highest levels of schooling completed by parents 


Immigration status of student 


Country of birth of student and parents 


Student learning styles 


Information on how student studies 


Student attitudes and reading activities 


Student preference and behaviour for specific 
reading activities 



CONTENTS OF THE OUESTIONNAIRES 

The extensive focus on the four levels of the educational system enables an informed search for statistical 
patterns that are linked to differences in achievement among countries, schools, classrooms, and individual 
students. But to reach this goal, the pertinent data that will inform these relationships is needed. PISA 2009 
obtains appropriate background information through two questionnaires, one for schools (filled in by the 
principal or designate) and one for students. These two survey instruments are administered in all participating 
countries and economies. In addition, three survey questionnaires are provided as international options that 
countries can elect to have administered to enrich the information base for their schools and students. 

■ School questionnaire 

■ Student questionnaire 

■ Parent questionnaire (international option) 

■ Educational career questionnaire (international option) 

■ Questionnaire on student familiarity with ICT or information and communication technology 
(international option) 

Annex B provides a complete list of the items of each questionnaire, but the purpose of this section is to present 
a brief summary of their contents. 

It is important to note that most background items sought in the questionnaires were also used in previous 
PISA surveys, establishing continuity of data collection for comparison and the ability to search for trends over 
time. Additional items were created to explore new theoretical and policy dimensions of the achievement 
relationship, and the new items were subject to field testing to ensure that they were understandable to the 
respondent and measured the intended information goal. The field testing led to reconsideration and refinement 
of some of the new measures and omission of those that did not seem to serve their purpose. 



PISA 2009 ASSESSMENT FRAMEWORK - KEY COMPETENCIES IN READING, MATHEMATICS AND SCIENCE O OECD 2009 



PISA 2009 QUESTIONNAIRE FRAMEWORK 



School questionnaire 

The respondent who is responsible for providing information on the school questionnaire is the school principal or 
designate, presumably the most knowledgeable person on school characteristics and functioning. This instrument 
was designed to take about 30 minutes to complete and is the key source of information on all dimensions of the 
school with additional information provided by students. Questionnaire items have been designed for transparency 
and to provide a format which simplifies its administration. It seeks to obtain a comprehensive picture with questions 
on the: 



■ structure and organisation of the school 

■ student and teacher body 

■ school's resources 

■ school's instruction, curriculum and assessment 

■ school climate 

■ school's policies and practices 

■ characteristics of the principal or designate 

Student questionnaire 

The student is responsible for providing the information in this questionnaire with an estimated 30 minutes for 
completion. Although obtaining more information in more detail from students would be desirable, the fact that 
they must also sit two hours of assessments is an important reason for limiting the length of the questionnaire. 
The specific categories of information that this questionnaire tries to assess are the student's: 

■ educational background 

■ family and home situation 

■ reading activities 

■ learning time 

■ school characteristics 

■ classroom and school climate 

■ language classes 

■ library access and activities 

■ strategies for reading and understanding texts 

Parent questionnaire (internationai option) 

This optional questionnaire is adopted by some countries and economies and delivered to parents by their 
1 5-year-old child. The purpose is to gain considerable additional information on both parents and their children 
that could be used for more detailed and deeper statistical analysis of achievement patterns. Particular detail 
is sought on: 

■ basic parent characteristics 

■ child's past reading engagement 

■ parents' own reading engagement 

■ home reading resources and support 

■ parents' background 

■ parents perceptions of and involvement in child's school 

■ extent of school choice 
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Questionnaire on educational career (international option) 

This brief questionnaire is comprised of only six student questions: 

■ extended absences 

■ frequency of changing schools 

■ expected educational attainment 

■ lessons or tutoring outside of school 

■ marks received in language 

Questionnaire on student familiarity with ICT (international option) 

This questionnaire is designed to ascertain a student's access to computer technologies as well as the extent of 
use and capabilities for different educational applications. Specific dimensions are: 

■ availability of ICT devices at home 

■ availability of ICT equipment at school 

■ use of computers for educational activities in school 

■ use of computers for educational activities outside of school 

■ student capability at computer tasks 

■ attitudes towards computer use 

The first two questionnaires, i.e. the student and school questionnaires, are used to provide background information 
to interpret the achievement results in all participating countries and economies. The last three can only be used 
for countries and economies which incorporated them as international options. However, the overall collection of 
detailed information on background variables at all levels of education makes it possible to inquire into particular 
policy themes which may contribute to better educational achievement in reading and other subjects. PISA 2009 
is dedicated to collecting empirical evidence and informing policy making in a variety of relevant areas. To 
illustrate the use of the questionnaire information for improving educational policy, the following section provides 
examples of the types of policy themes that might be chosen for in-depth investigation using PISA results. 



INFORMATION FOR IN-DEPTH INVESTIGATIONS 

Based upon what has been learned from the analysis of earlier PISA studies, the information collected to assist in 
the interpretation of student performance in PISA 2009 has been enhanced in two ways. First, the accumulated 
experience acquired from the earlier PISA studies and the more recent scientific progress (Bransford, 2000) on the 
knowledge base for learning has informed which variables and questions are most essential for further analysis. More 
specifically, the PISA 2009 questionnaires attempt to capture background dimensions affecting reading performance 
that are reflected in the increasingly sophisticated literature on reading development that has emerged since the first 
effort to address reading in PISA 2000. Second, the increased focus on using PISA results to improve educational 
policy has identified several specific policy themes for which it is hoped that PISA 2009 data might be informative. 

PISA 2009 will provide a rich data base to seek information on a variety of topics that are of importance to 
individual countries or provide an overall assessment of educational performance patterns across countries. 
Among the many policy themes that could be addressed, five are outlined below. Although considerable 
information has been collected in previous PISA cycles that bear on these themes, a special effort has been 
made in PISA 2009 to gain more refined detail about: 

■ system level indicators 

■ effective learning environments in reading 

■ school effectiveness and school management 

■ educational equity 

■ cost effectiveness 



PISA 2009 ASSESSMENT FRAMEWORK - KEY COMPETENCIES IN READING, MATHEMATICS AND SCIENCE O OECD 2009 



PISA 2009 QUESTIONNAIRE FRAMEWORK 



For each of these themes an attempt has been made to identify specific policy concerns that might be the focus 
of subsequent inquiry and to provide additional information that might help reply to those questions. 

System level indicators 

Most analyses addressing patterns of educational performance have focused on differences in features among 
individual students, classrooms, and schools that contribute to educational achievement. This is understandable 
because a plethora of educational studies shows that differences in instruction, curriculum, and school and 
home resources account for important differences in educational experiences and achievement results. 
However, national, federal or regional governments have relatively limited influence on the finer details of 
what happens in homes and schools. Rather, they are able to establish the larger organisational and resource 
framework in which education takes place and that might ultimately influence school and family behaviour. In 
particular, there are three dimensions at the system level which are both an active topic of educational policy 
and which might affect the educational performance at the system level. These are the degree of centralisation 
or decentralisation of decision-making authority, evaluation and accountability, and the degree to which 
secondary education is comprehensive or specialised. 

In recent years many countries have been addressing the issue of how their schools can be more responsive 
to the needs of specific students, groups of students, and communities. When decision-making is vested in a 
central authority such as the national government, it is argued that teacher hiring and training and curriculum 
are too rigid to accommodate the types of differences in student learning found in the schools, and especially in 
countries with different social conditions and ethnicities. For this reason, many nations have established a variety 
of reforms to decentralise their schools that place educational decisions closer to the students and communities 
being served with the hope that this will improve school performance (Hannaway & Carnoy, 1 993). Questions 
on centralisation and decentralisation of school finances, curriculum, teacher staffing, and other areas of school 
operations have been included in previous PISA surveys, but for PISA 2009 more decision areas are addressed 
to enable analysis of the impacts of different types of decentralisation on student performance. 

Different forms of educational evaluation and accountability have also become more prominent over time. 
Educational entities at all levels from students and their families to classrooms, schools, regions and nations 
wish to ascertain the degree to which they are experiencing educational success. At the same time the 
measurement of educational outcomes provides indicators of accountability of the different decision-making 
entities in education. That is, the placement of authority and responsibility for decision-making at any particular 
level implies responsibility for results at that level, and the concern for accountability. 

Countries and their regions have established different approaches to evaluation and accountability. Some 
sponsor periodic systems of testing to gain standardised information on student and school performance at 
national, federal and highly decentralised levels. Others require such evaluations, but leave it to the regional 
and local authorities to carry out testing and assessment. Others however lack systematic evaluations. But, 
accountability also extends to the consequences carried by evaluation results (Koretz, 2008; Carnoy, Elmore & 
Siskin, 2003). Some countries simply use their surveys of educational performance to inform the general public 
and the various educational entities. Others use it to provide rewards to schools and teachers for good results 
and create incentives for better performance. Evaluations can also include school inspections and various forms 
of individual schools' evaluation activities. They can be used to fix educational performance objectives for 
students. Eor example, end-of-course or other external exams might be required for course credit or graduation. 
Other countries add resources and technical assistance to improve schools where performance is low. The 
information on evaluation and accountability collected in PISA 2009 has been refined to consider the types of 
approaches that have been adopted and their possible consequences. 

A much bigger debate has taken place over the structure of secondary education. Traditionally countries 
divided secondary schools between those with an academic focus and those with a vocational, professional, 
or technical focus. Academic secondary schools were designed to accommodate students who were preparing 
for further study at university. The other types of schools focused on students who would be entering the labour 
force directly or undertaking work apprenticeships or short courses to prepare for life in the labour force. At 
the end of secondary school, students would be divided between the two types of schools according to both 
their academic performance and their preferences or orientations. Typically academic secondary schools would 
choose only the higher performing students. 
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A historic debate has taken place within and among countries on whether such educational separation at a 
relatively early age denies academic opportunities for those who develop intellectually at a later age, and 
whether opportunities are determined according to the socio-economic status of the student's family with 
those of lower status relegated to the vocational institutions (Levin, 1978). Accordingly, many countries have 
considered establishing and maintaining comprehensive high schools that offer both academic and vocational 
courses where all students are required to undertake at least a common core of academic work. Even within 
such schools there are debates about whether academic classes should be heterogeneous in enrolments or 
streamed by ability. Since students enter educational systems characterised by different structures regarding 
comprehensiveness of curriculum and student enrolments, it is useful to provide measures of these differences 
to ascertain both their patterns and potential impact on achievement. Information is gathered in PISA 2009 with 
the goal of relating it to both overall achievement levels as well as equity in the distribution of achievement. 



An in-depth report on this subject would pose a range of research questions and statistical methods to evaluate 
the potential impacts of decentralisation, evaluation and accountability, and secondary school structure. They 
would be constructed to explore the impact of these types of arrangements on the achievement of educational 
systems, schools, classrooms, and individual students. Whether causal relations can be inferred may be 
problematic, but consistency of patterns should be identified for further investigation. 



Effective learning environments in reading 

A great deal of research on learning effectiveness in reading has been undertaken since the publication of the 
PISA 2000 results which focused on reading (Kamil, 2010). In part this research activity was induced by the 
PISA reading results and the data that were made available in that report on opportunities to learn and improve 
reading. There have also been more general attempts in recent years to both produce and summarise the results 
on how to create reading success. All of the previous PISA surveys have gathered considerable information 
on the instructional strategies, curriculum, teaching resources, and home opportunities for enhancing student 
achievement. While not all of these survey questions are pertinent strictly to reading, it should be noted that 
even mathematics and science build on capable reading skills. 

A major shift has taken place in the last decade in the student populations that are of interest for studying 
reading achievement. Most research on reading in the past was focused on the elementary school on the 
assumption that reading is so fundamental to further learning that it is considered to be a basic skill that must 
be acquired in the early years of schooling. Thus adolescents were considered to have developed their reading 
skills in the elementary school, and reading was not a subject of high priority at the secondary level. 

In this respect two OECD surveys have shifted much of the focus to the entire educational system. The OECD's 
International Assessment of Adult Literacy Survey (lALS) found that many adults lacked fundamental literacy skills 
in understanding reading and applied mathematics, even among those who had attended or completed secondary 
school. But more recently, and largely as a result of the findings of PISA 2000, it became recognised that a 
substantial number of 15-year-olds are not proficient readers for the knowledge society. It seems reasonable to 
assume that reading still is a basic skill which should be acquired in early years, but this skill needs to be further 
supported and developed throughout life. Students who for some reason do not acquire good reading skills early 
in elementary school are disadvantaged in many ways in comparison to those who obtained those skills early on. 
Thus, the challenges of reading performance at the secondary level have been widely confirmed and have become 
topics of educational concern. This concern has been exacerbated by the growth of immigrant populations whose 
mother tongue differs from the official language of their adopted society and schools. But, even among native 
students, PISA 2000 and individual country studies have raised concerns about the status of struggling adolescent 
readers, ones who can, perhaps, identify and pronounce most words and sentences, but who have difficulties fully 
understanding content and meaning of what they are able to decipher (Deshler et al., 2007). Because reading is 
so fundamental to overall literacy and the learning of the other subjects, poor reading performance is an obstacle, 
in itself, to secondary school academic achievement and further education. 

Accordingly, an in-depth investigation of effective learning environments in reading would collect information 
in PISA 2009 that would enable further research and study for improving the conditions to advance reading 
performance. That report would contain considerable insights into teaching and learning conditions and their 
possible relationships to achievement in reading and other subjects. It would enable a deeper range of research 
that would explore descriptive patterns as well as those that come closer to attempts to model the relationships 
in a causal manner and to test various hypotheses about the conditions that lead to greater achievement. 
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This report would address the influence of variables at the level of the system, the school, the classroom, 
and the individual student and family environment by drawing on the considerable information collected 
at each level. What is particularly unique about the approach in PISA 2009 is that it advances the types of 
background and explanatory information collected in the earlier survey on reading for PISA 2000 by building 
on subsequent research. Thus, it adds more information on teachers that has been identified as potentially 
pertinent to learning. It places greater emphasis on students' higher order skills, self-regulated learning, and 
constructivist approaches to learning, all which have been investigated and found to show great promise 
in raising student understanding and knowledge. It also places more stress on learning facilitation and the 
student role in learning, including levels of student interest, motivation, and engagement. And in considering 
the opportunity to learn, it explores more fully the content of reading activities at home and at school. 

To address policy relevant concerns, an in-depth report on effective learning environments in reading could 
incorporate questionnaire items on teacher characteristics such as professional knowledge, teaching motivation, 
and instructional methods; on classroom characteristics such as class size, student composition, and classroom 
climate; and the teaching processes that are incorporated into the learning environment. In addition, it could 
take account of background variables on students such as socio-economic, immigrant status, gender, and their 
learning approaches and attitudes towards learning. An attempt could be made to uncover both salient patterns 
of relations between these variables, many of which can be altered or partially changed, and the apparent 
achievement results. 

School effectiveness and school management 

A considerable amount of research has been carried out in an attempt to differentiate the characteristics of 
effective schools from those of ineffective schools. The purpose is to identify the characteristics of schools that 
are effective in raising student achievement and to apply this knowledge to improve schools. Certainly, this is 
a potentially important application of information collected by PISA 2009 for guiding school improvement in 
the participating countries and economies. However, it is important to note that PISA data only captures the 
situation of students at one point in time in a year. 

In general, research on effective schools develops statistical contrasts among successful schools enrolling 
students of lower socio-economic status in the same geographical area with schools that are not successful 
academically. Schools with considerably higher academic achievement for the same types of students are 
considered to be effective schools , and those with poor results are considered to be ineffective schools . 
The two groups of schools are then compared for their operational features and resources to ascertain how they 
seem to differ in such areas as goals, leadership, teachers, curriculum, organisational climate, instruction, and 
evaluation. Substantial differences along these dimensions are noted between effective and ineffective schools 
serving the same types of students, and these effective school dimensions become the focus of policy attempts 
to improve schools (Scheerens, 2000;Teddlie & Reynolds, 2000). 

Research on effective schools using PISA 2009 data can help to identify potential features of schools that have 
shown previous connections to student achievement. An in-depth report on school effectiveness and school 
management could attempt to include information on all of these effectiveness dimensions in studies of the 
determinants of educational achievement for PISA 2009. That is, using the effective school correlates as a 
guide, it could construct variables and potential relationships to attempt to explain statistically the differences 
in student achievement among and within countries. Since many of these variables will be ones amenable to 
policy manipulation, it should be possible to suggest generally among countries and for individual countries the 
most promising policy levers for raising achievement. 

A particularly important feature of effective schools is that of the school leadership. Virtually all studies of 
effective schools find that school leadership is central to school effectiveness and identify specific features of such 
leadership that appear to be particularly important. Effective school leaders put particular effort into establishing 
and maintaining academic goals among students and school staff as well as maintaining a safe and orderly 
environment and providing the teaching support and instructional policies that create academic growth among 
students. They are also attentive to evaluating results and using evaluations to improve the school operations. 

The PISA 2009 questionnaires contain considerable data that would be useful for assessing both school effectiveness 
and school management and leadership. For example, they reflect questions on the specific leadership activities 
undertaken by the principals of participating schools, activities that have been linked to student achievement 
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(Hallinger & Heck, 1998). Thus, in addition to the overall attempt to discern policy influences of schools on 
achievement, a special effort could be devoted to that of school management and leadership. 

Since school effectiveness research attempts to either uncover or infer causal relations between school 
characteristics and student achievement, there are a number of challenges that must be undertaken in 
PISA 2009 in order to be able to provide meaningful outcomes. One challenge is that significant numbers of 
1 5-year-olds among countries have just begun to attend secondary schools or have attended them for only one 
or two years. Thus, their present achievement as measured in PISA 2009 is likely to be more closely connected 
to their previous years of schooling than to the characteristics of their present schools. However, this can be 
partially addressed by choosing countries for the analysis where the students have attended their present schools 
for many years. In addition to the findings that are developed from PISA 2009 on effective schooling and school 
management directly, it is likely that this report could develop a future design for value-added measures where 
prior achievement is accounted for in exploring how much achievement takes place in the present school. 

Educational equity 

A central pre-occupation of PISA is that of fairness or justice in access to education and the opportunity to 
learn. Virtually all countries participating in PISA 2009 recognise that the fullest development of all of their 
human resources possible is crucial to a just and prosperous society. An in-depth investigation of educational 
equity and equality requires delineation of two criteria (Coleman, 1993). The first criterion is the identification 
of population subunits that are to be scrutinised for equity. Although different societies may identify different 
groups for comparative study, the common population categories that are often used to study equity in education 
are: gender, socio-economic status, race, religion, ethnicity, degree and type of disability, immigrant status, 
language, geography (e.g. regions or urban/rural distinctions), and public/private school enrolment. 

The second concern is that of how to measure equity. Equity can be assessed in terms of the distribution 
of access to schooling, learning resources and opportunities, and educational outcomes. The richness of 
PISA 2009 is that most of the population comparison categories are available in the data, and versions of each 
equity criterion can be used for investigations on educational equity. Other than religion, ethnicity, and detailed 
data on disabilities, the background questionnaires collect information on the other sub-population groups as 
well as abundant measures of schooling characteristics and achievement outcomes in all three subjects. 

Analyses could be done on linking school effectiveness variables to their distribution. For example, home 
opportunities, school organisation, instructional strategies, curriculum, class size, and other opportunity-to- 
learn variables, leadership, and teacher characteristics that have been linked to student achievement could 
be evaluated for their distribution across different sub-populations of students. Thus, these variables could 
be examined not only for their apparent influence on the overall level of achievement, but also on equity 
in their own distribution and the implications for equity in achievement. Specific resources that are used 
for compensatory education, that is to compensate for educational disadvantage, could be assessed for their 
distribution across needy populations (Field, Kuczera & Pont, 2007). Analyses could also be undertaken to 
ascertain if there is a loss of effectiveness when equity policies are implemented, that is, if overall school 
effectiveness suffers when inequities are addressed. Of particular policy importance, public interventions for 
reducing inequities such as adding resources to classrooms and schools and providing other compensatory 
policies could be evaluated for their efficacy at improving equity. And, a specific attempt could be made to 
ascertain the impact of student composition or segregation of student subgroups on educational performance, 
a topic of great currency for addressing equity in education. 

Finally, an in-depth investigation on this issue would represent a unique opportunity to explore how educational 
equity has changed since 2000. The previous PISA reports contain many of the same background questions 
found in PISA 2009. This means that trends in educational resources, opportunities, and outcomes could be 
assessed for countries as a whole as well as for their subpopulations. In particular, socio-economic and gender 
differences in achievement results could be compared with changes in educational policy as evidenced by the 
responses to similar questions that have been used in the four administrations of the PISA studies from 2000 to 
2009. This type of longitudinal analysis could also be carried out for other subgroups of students. Thus, such an 
in-depth investigation would be able to provide concrete information on how equity has changed both among 
and within countries over almost a decade. 
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Cost-effectiveness 

All educational systems are constrained by available resources. In the quest for improving both levels of academic 
performance and educational equity, choices must be made among different educational policies in terms of their 
contributions to effectiveness. But, such choices also have important consequences for costs. That is, there may 
be considerable differences in costs of educational strategies with similar effectiveness requiring larger budgets 
and greater resources for some strategies of equal effectiveness than others. Cost-effectiveness analysis takes into 
account not only the educational impact of any particular approach, but also its costs (Levin & McEwan, 2001). 
Its application enables the adoption of educational practices that provide the greatest improvement relative to their 
cost by accounting for both the effectiveness of different alternatives and the resources that they require. In so doing, 
all resources can be used most efficiently and a given budget and other resources not reflected in the budget can 
produce the largest possible gain in student achievement and equity consistent with resource constraints. 

The effectiveness component of the cost-effectiveness method could be derived from the previous analyses of 
school effectiveness and effective learning environments, as outlined above. Equity results could be summarised 
as outlined above. Cost-effectiveness analysis combines this information on effectiveness of alternatives with 
costs to compare the productivity of alternatives relative to their resource requirements. Resource measures 
include the value of both public and private costs, measures which are not normally summarised in government 
expenditures. Public budgets use their own accounting practices which often distort costs of government 
outlays and ignore costs paid by families, other government agencies, and non-governmental organisations. 
Accordingly, the most appropriate method for estimating costs according to economists is to set out the specific 
personnel, facilities, and other resources that are required for each alternative and to estimate their costs 
directly, a full cost accounting process known as the ingredients or resource method. 

An in-depth investigation of cost-effectiveness could then begin with a summary of the achievement and equity 
consequences of different policy alternatives and a description of the resources required for each one. Data on 
costs of programmes is not collected in PISA 2009, so the data would have to be collected independently for 
each country included in the analysis. The total costs of each alternative that has shown educational advantages 
in raising achievement or improving equity would then be estimated and compared with the magnitude of 
expected educational results. These comparisons would allow a ranking of alternatives for raising achievement 
or providing greater equity by their efficiency in the use of resources. 

It must be noted that although such an analysis can be done among countries, the averages in terms of both 
effectiveness and costs tend to hide important differences in both costs and effectiveness that are unique to 
individual countries. Further, educational policy is set on a country-by-country basis, so that it makes sense to 
carry out the cost-effectiveness analysis for each country to guide its policy decisions. 

Cost-effectiveness analysis can be applied to any policy which yields effectiveness results. For example, to 
the degree that decentralisation, accountability, school structure, school choice, teacher selection, class size, 
curriculum, instructional strategies, desegregation, and other educational policies are associated with improved 
school and student performance, it is possible to match the results for such improvement with the costs of 
obtaining the gains. Then the educational outcomes relative to their costs could be compared for each of the 
feasible policy alternatives. The goals of an in-depth investigation on cost-effectiveness would be to elaborate 
on the method and apply it to the analytic results on effectiveness, relying primarily on information collected 
for PISA 2009 in the questionnaires. 

The systematic attempt to use the research findings on learning and conceptual linkages in interpreting PISA 
2009 achievement results requires considerable background information on educational systems, schools, 
classrooms, and students and their families. Obtaining this information is made possible through the survey 
questionnaires which have been carefully designed to collect the pertinent interpretive information. Through 
this most comprehensive approach to pertinent data collection, PISA 2009 will make its most extensive effort 
to address the policy inferences and consequences of its findings. 
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AN N EX A1 

Print reading sample tasks 



The print reading sample tasks are examples of questions students 
answered in the PISA 2009 survey to assess their competencies in 
reading printed text. 

Note that the numbering of the questions is identical to the 
numbering used in the test booklets given to students. 
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Macondo 

Dazzled by so many and such marvellous inventions, the people of Macondo did not know where 
their amazement began. They stayed up all night looking at the paie electric bulbs fed by the plant 
that Aureliano Triste had brought back when the train made its second trip, and it took time and effort 
for them to grow accustomed to its obsessive toom-toom. They became indignant over the living 
images that the prosperous merchant Don Bruno Crespi projected in the theatre with the lion-head 
ticket windows, for a character who had died and was buried in one film, and for whose misfortune 
tears of affiiction had been shed, would reappear alive and transformed into an Arab in the next 
one. The audience, who paid two centavos apiece to share the difficulties of the actors, would not 
tolerate that outlandish fraud and they broke up the seats. The mayor, at the urging of Don Bruno 
Crespi, expiained by means of a proclamation that the cinema was a machine of iilusions that did not 
merit the emotional outburst of the audience. With that discouraging explanation many feit that they 
had been the victims of some new and showy gypsy business and they decided not to return to the 
movies, considering that they already had too many troubles of their own to weep over the acted-out 
misfortunes of imaginary beings. 



Macondo is a piece of prose from the novel One Hundred Years of Solitude by the Colombian author 
Gabriei Garcia Marquez. It is classified as beionging to the personal situation because it was written for 
readers' interest and pieasure. The Macondo unit in PISA is introduced with a brief paragraph to orientate 
the reader: "The passage on the opposite page is from a novei. In this part of the story, the raiiway and 
electricity have just been introduced to the fictionai town of Macondo, and the first cinema has opened." 
The people's reaction to the cinema is the focus of the passage. Whiie the historical and geographical 
setting of the extract is exotic for most readers, going to the movies is within the experience of 1 5-year- 
olds, and the characters' responses are at the same time intriguing and humanly familiar. Within the 
continuous text format category, Macondo is an example of narrative writing in that it shows, in a 
manner typical of this text type, why characters in stories behave as they do. It records actions and events 
from the point of view of subjective impressions. 



Question 1: MACONDO 

Whgt feature ofthe movies cguse4 the people of Mgcondo to become gngryi’ 
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Framework Characteristics 



Situation 


Personal 


Medium 


Print 


Text format 


Continuous 


Text type 


Narration 


Aspect 


Integrate and interpret: Develop an interpretation 


Question intent 


Infer a reason for characters' behaviour 


Item format 


Open constructed response 



CODING GUIDE 

Code 2: Refers to the fictional nature of movies or more specifically to actors reappearing after they 

have "died". May quote directly from the third sentence ("...a character who had died and 
was buried in one film, and for whose misfortune tears of affliction had been shed, would 
reappear alive and transformed into an Arab in the next one ...") or the last phrase 
("the acted-out misfortunes of imaginary beings."). 

■ People who they think have died came back to life. 

■ They expected the films to be true and they aren't. 

■ They think that the man in the film has pretended to die, and that they are being taken for fools. 

■ A character who had died and was burled in one film would reappear alive in the next one. 

■ They don't understand that the films are fiction. 

■ Because actors whose characters had died In the previous film returned as new characters in the 
next film. The audience felt they were being robbed of their emotions. [Elements of both 2 and I .] 

■ They thought they had enough problems of their own without watching pretend people act out 
problems. [Clear understanding of the part "fiction" plays in the people's anger, though has taken 
it a stage further.[ 

■ Because one of the actors was buried in the film and he came back as an Arab. 

[Marginal: very specific. [ 

Partial credit 

Code I : Refers to fraud or trickery, or to the audience's thwarted expectations. May quote directly 

"that outlandish fraud" or "the victims of some new and showy gypsy business". 

■ They think they are being defrauded. 

■ Because they feel they have wasted their emotions for nothing. 

■ They felt they had been the victims of some new and showy gypsy business. 

■ They would not tolerate that outlandish fraud. 

■ Because they paid 2 centavos apiece for that outlandish fraud they could not tolerate. [Partly 
direct quotation from lines 9- 1 0; not specific reference to the nature of the fraud. [ 

■ Because they were unaware of the workings of it. [Take as a way of saying "fraud" (general).) 

No credit 

Code 0: Gives insufficient or vague answer. 

■ They were angry with Bruno Crespi. 

■ They didn't like the films. 

■ They want to get their money back. 
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■ They thought they had been victims. 

■ They were violent. 

■ They were stupid. 

■ They are expressing their feelings. 

■ They paid 2 centavos and didn't get what they wanted. ["What they wanted" is too vague.) 

OR: Shows inaccurate comprehension of the material or gives an implausible or Irrelevant answer. 

■ They felt they shouldn't be bothered with other people's problems. [Wrong: the people DID want 
to be bothered with REWL peoples problems.] 

■ It's the only way they have of protesting at wasting their money. 

■ They were angry because they had to see a person that was dead and buried. [The quoted 
section gives the sense of "they did not like seeing dead people in films" - incorrect interpretation.] 

Code 9: Missing. 



The text positions the reader at a distance - albeit a sympathetic distance - from the people of Macondo. 
To gain full credit for this item, it is necessary to stand outside the text and recognise the difficulty these 
unsophisticated villagers have in coming to grips with the difference between fiction and real life. For 
most modern readers, this difficulty would seem unusual, and contrary to expectations. Just over half of 
the students in the field trial were able to demonstrate that they could develop this interpretation. The 
partial credit response requires a less challenging imaginative act on the reader's part: understanding that 
people get angry if they feel that they are being tricked is, for most people, quite easy to imagine. This 
kind of answer gains partial credit because it is not wrong - but it only tells part of the story. Although 
a relatively small percentage of students in the field trial gave this kind of response, their overall reading 
ability was substantially lower than that of those who gained full credit. 



Question 3: MACONDO 

At the en4 of the pqssqge, why 4i4 the people of Mqcon4o 4ed4e not to return to the moviesi’ 

A. They wqnte4 amusement qn4 4istrqction, but foun4 thqt the movies were realistic 
qn4 4epressing. 

B. They coul4 not qffor4 the ticket prices. 

C. They wqnte4 to sqve their emotions for reql-Iife occasions. 

P. They were seeking emotional involvement, but foun4 the movies boring, 
unconvincing qn4 of poor quality. 
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Framework Characteristics 



Situation 


Personal 


Medium 


Print 


Text format 


Continuous 


Text type 


Narration 


Aspect 


Integrate and Interpret: Form a broad understanding 


Question intent 


Infer motives for characters' actions 


Item format 


Multiple choice 



CODING GUIDE 

Full credit 

Code I : C. They wanted to save their emotions for real-life occasions. 

No credit 

Code 0: Other. 

Code 9: Missing. 



This task requires integrating and interpreting to form a broad understanding. In order to gain credit, 
students need to synthesise elements across the text to identify the reason that the characters in the story 
behaved as they did at the end. In selecting alternative C, they must reject some reasons that could 
plausibly explain why people might decide not to go the movies, represented by distractors that are 
based on preconceptions rather than on the text. 



Question 4: MACONDO 

who gre the "imggin^ty beings" referred to in the Igst line of the p^ss^gei” 

A. Ghosts. 

B. Fairground inventions. 

C. Chgr^cters in the movies. 

D. Actors. 
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Framework Characteristics 



Situation 


Personal 


Medium 


Print 


Text format 


Continuous 


Text type 


Narration 


Aspect 


Integrate and interpret: Develop an interpretation 


Question intent 


Interpret a referent when the antecedent is implied rather than stated 


Item format 


Multiple choice 



CODING GUIDE 

Full credit 

Code I : C. Characters in the movies. 

No credit 

Code 0: Other. 

Code 9: Missing. 



To gain credit for this item, students need to follow a chain of references that begins about one-third of 
the way into the passage. Almost three quarters of the students who took part in the field trial were able 
to identify the "imaginary beings" with the characters who are first referred to as "living images". The 
need to make connections across the text explains the item's classification in the integrate and interpret 
aspect category. Most of those who did not select the correct alternative chose option D, apparently 
confusing the fictional with the real. 



Question 5: MACONDO 

Do you ggree with the fingl ju4gement of the people of Mgcondo gbout the vglue of the moviesi’ 
ExpUin your answer by comparing your gttitu4e to the movies with theirs. 
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Framework Characteristics 



Situation 


Personal 


Medium 


Print 


Text format 


Continuous 


Text type 


Narration 


/\spect 


Reflect and evaluate: Reflect on and evaluate the content of a text 


Question intent 


Compare characters' attitudes with personal knowledge and experience 


Item format 


Multipie choice 



CODING GUIDE 

Full credit 

Code I : Refers to attitudes to "realism" and/or emotional invoivement in the movies. Answer must be 

consistent with the idea that the people of Macondo are looking for realism in the movies. 
Comparison between Macondo and personai experience/attitude may be impiied or stated. 

■ Once you realise it's not real you can just use the cinema for escapism. ThereS no need to get so 
involved in the lives of the characters. 

■ Yes, I agree, there's enough suffering in the world without making it up. 

■ No, people understand that when you go to the movies what happens on the screen isn't real. 

■ Unlike the Macondo people, I can cry my eyes out during the film, but I forget about it when I 
leave the cinema. 

■ I agree with them. Why do people want to upset themselves about movies? That's why I'm doing 
science, because it's about fact not fantasy. 

■ I enjoy the movies because it makes me focus on other people's problems and not mine. 

■ It depends. If the film is rubbish I feel like walking out, but if itk good you are caught up in it and 
don't mind that ifi not true. 

■ No, I enjoy movies for entertainment. 

■ Yes the movies are just made up. lU way better when you see people acting things out live. 

■ No, in movies events are usually exaggerated. 

■ I do not agree with their reactions because movies are a form of entertainment and are not to be 
taken too seriously. However the people of Macondo do not know any better and so I understand 
how they would feel. 

OR: Refers to social, historical or cultural context, for example in terms of comparative familiarity 

with technology, changes in social experience. Answer must be consistent with the idea that 
the people of Macondo are looking for realism in the movies. Comparison between Macondo 
and personai experience/attitude may be impiied or stated. 

■ The people of Macondo were primitive and reacted emotionally. I and most other people 
nowadays are more sophisticated. 

■ They started to watch the films from a wrong position. They didn't understand that this was not 
the news, but entertainment. From that point of view their reaction is understandable. Films are of 
course to be seen as entertainment. That is the point with them. 

■ People these days don't get upset by films. 

■ Yes, I would agree if I were one of them because they had never seen it before. 



PISA 2009 ASSESSMENT FRAMEWORK - KEY COMPETENCIES IN READING, MATHEMATICS AND SCIENCE © OECD 2009 



179 



ANNEX A1 • PRINT READING SAMPLE TASKS 



No credit 

Code 0: Gives insufficient or vague answer. 

■ I'm like the people of Macondo because I often think that the movies are a waste of time. 

■ I love the movies. I can't understand their reaction. 

OR: Shows inaccurate comprehension of the material or gives an implausible or irrelevant answer 

■ Yes, it seems the movies, rather than being relaxing and easy to watch, were realistic and 
emotional. Movies are a form of entertainment, a scape-goat and a way of forgetting the troubles 
of real-life for a laugh. The people of Macondo were disappointed because the movies they saw 
were not relaxing but rather quite full-on and they wanted something that would entertain them. 
[First two sentences by themselves are not relevant to the question. Last sentence indicates a 
misunderstanding of the text.] 

■ No, they should be more expensive and have cup holders, popcorn, frozen coke and lollies. The arm 
rests should lift up, footrests, surround sound. [Maybe a joke, but if not - a misreading of the text.] 

■ These days there are laws against damaging property in cinemas. [Answer focuses on behaviour 
rather than attitude.) 

■ We have better cinemas now. [Irrelevant] 

■ Yes because the movie wasn't very good and would have made them nervous. [Misunderstood 
the question.] 

Code 9: Missing. 



This is a good example of a reflect and evaluate item that asks readers to draw on their own experience 
and ideas and compare them with those provided in the text. To gain credit for this item students 
must assess the views attributed to the people of Macondo against their own knowledge of the world, 
providing some evidence that they have an understanding of what is said and intended in the text. While 
this kind of item does not demand critical evaluation in the sense of applying any kind of formal analysis 
or reasoning, it does require that readers engage with the narrative to the extent that they compare 
or contrast its content with personal beliefs. The coding guide indicates that contradictory responses 
can gain full credit as long as there Is evidence of understanding of the issue in focus, and of their own 
reflection (for example, compare "Yes, I agree, there's enough suffering in the world without making it 
up" and "No, people understand that when you go to the movies what happens on the screen Isn't real"). 
About half of the students In the field trial sample gained credit for this item. 
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PRINT READING UNIT 2 



Library Map 



EXIT 




The library map that forms the basis of this unit is an example of a kind of everyday non-continuous text 
that is often encountered in work, personal, public and educational settings. The context of this example 
is defined as public because the map relates to the activities of a community (a public library) and assumes 
anonymous contact with the reader. In terms of text type, the map is classified as description, since the 
information it contains refers to properties of objects in space and their relationship to one another. 
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Question 5: LIBRARY 

Fof school you need to fegd 3 novel in Ffencb. On the mgp 4f9w 9 circle gfound the section wherc 
you would be most likely to find 9 suitable book to boffow. 



Framework Characteristics 



situation 


Public 


Medium 


Print 


Text format 


Non-continuous 


Text type 


Description 


Aspect 


Access and retrieve: Retrieve information 


Question intent 


Locate information that matches on one factor using low-level inference 


Item format 


Short response 



CODING GUIDE 

Full credit 

Code I : Circles the words "other languages" or the lines (shelves) near the words. 







PISA 2009 ASSESSMENT FRAMEWORK - KEY COMPETENCIES IN READING, MATHEMATICS AND SCIENCE © OECD 2009 






ANNEX A1 • PRINT READING SAMPLE TASKS 



No credit 

Code 0: Other, including circling which includes any other feature of the map completely. 







Code 9: Missing. 



This short response item requires that the reader search for, locate and select relevant information from the 
information space: in this case, a map. The required information is found in a single location rather than 
multiple locations, a factor that is likely to reduce difficulty. On the other hand, the match between the 
words in the task and the caption on the map is not literal: the reader must make an inference to categorise 
"French" as "Other languages". (A translation and adaptation note instructed that in national versions of 
the item the language referred to in the item should be a foreign language commonly taught in schools.) 
Nevertheless, this is a rather easy item, with more than four fifths of the students in the field trial able to 
identify the right section of the library. As indicated in the full credit examples provided with the coding 
guide, students could mark the text in a number of different ways to show their answer. Although the 
question specifies that a circle should be drawn to show the answer, the format of the response is not the 
critical criterion for awarding credit: what is critical is whether or not the response clearly meets the intent of 
the question - "locating information that matches on one factor using low-level inference". 
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Question 7A: LIBRARY 

where gre New books locgtecji’ 

A. In the fiction section. 

B. In the non-fiction section. 

C. Negf the entfgnce. 

D. Negi" the infofm^tion desk. 

Library scoring 7a 

Note: The correct answer is C: "Near the entrance". This question is for information only and will not independently 

contribute to the student's score. The answer is taken into account in assessing the response to Question 7B. 



Question 7B: LIBRARY 

Explain why this location might hgve been chosen for New books. 



Framework Characteristics 



Situation 


Public 


Medium 


Print 


Text format 


Non-continuous 


Text type 


Description 


Aspect 


Reflect and evaluate: Reflect on and evaluate the content of a text 


Question intent 


Hypothesise about the location of a feature of a map drawing on personal 
knowledge and experience 


Item format 


Open constructed response 



CODING GUIDE 

Full credit 

Code 2: Answer to Part A correct. Gives an explanation which is consistent with the answer 

"near the entrance". 

■ People will see them as soon as they walk in. 

■ They are away from the other books, and people will find them easily. 

■ So people can look at them first. [Implies recognition that the new books are near the entrance.] 

■ So they are very visible. 

■ They are clearly visible and not hidden away among the bookshelves so that you have to search 
for them. 

■ You pass it on your way to fiction. 
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OR: Answer to previous question Part A correct. Gives an explanation which shows understanding 

of the location of the new books in relation to a part of the library other than the entrance. 

■ It gives children a chance to play while adults look around. [Recognises that the new books are 
near the Toys section.] 

■ When people are returning books they will see the new ones. 

Partial credit 

Code I : Answer to Part A incorrect. Gives an explanation which is consistent with the answer given 

for previous question. 

■ [Answer to Part A: In the fiction section.] Because this is the part of the library that most people 
would be using, so they would notice the new books. 

■ [Answer to Part A: Near the information desk.] Because they are next to the Information Desk, the 
librarian can answer questions about them. 

No credit 

Code 0: Gives insufficient or vague explanation regardless of whether answer to Part A is correct or 

incorrect. 

■ Because it's the best place. 

■ They are near the entrance too. [States where the new books are, without offering explanation.] 

■ The New books are near the suggestion box. [States where the new books are, without offering 
explanation.] 

OR: Shows inaccurate comprehension of the material or gives an implausible or irrelevant 

explanation, regardless of whether answer to Part A is correct or incorrect. 

■ So people would notice them when they were looking at the newspapers. [Inaccurate-implies 
that new books are near the newspapers.] 

■ Because there is nowhere else to put them, [implausible] 

■ Some people like to read new books. [Answer is irrelevant to question.] 

■ [Answer to Part A: In the fiction section.] So that they are easy to find, [answer irrelevant to 
answer given for Part A] 

Code 9: Missing. 



The coding rules for this task are somewhat complicated. Students are asked two questions - one 
multiple-choice and one constructed response - but only the second of these is coded directly. As this 
task contributes to the Reflect and evaluate scale, the multiple-choice component, which predominantiy 
requires retrieval of information, does not earn any credit on Its own. However, the multiple-choice 
question is taken into account in the coding of the second, constructed response question. 

To gain full credit, the response must include both accurate reading of the map (locating the New books 
near to the entrance) and a hypothesis about the reason for locating the New books in that position. 

To make such an hypothesis, readers need to consult their own experience or knowledge - in this case 
about the way libraries work and the way they are used by the public. In the PISA context, the outside 
knowledge required is intended to be within the expected range of 1 5-year-olds' experiences. 

Students receive only partial credit if they have failed to correctly locate the New books on the map, but 
have given a plausible hypothesis about the reason for locating New books in a particular position. Like 
the full credit responses, this kind of response fulfils the Intent of reflecting on content that is the main 
thrust of this task. 

This was an easy item, with over four fifths of the students in the field trial gaining full credit. 
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Student Opinions 



There are so many people out there dying from hunger and disease, 
yet we are more concerned about future advancements. We are leaving 
these people behind as we attempt to forget and move on. Billions 
of dollars are poured into space research by large companies each 
year. If the money spent on space exploration was used to benefit the 
needy and not the greedy, the suffering of millions of peopie could 
be alleviated. 



The chalienge of exploring space is a source of inspiration for many 
people. For thousands of years we have been dreaming of the heavens, 
ionging to reach out and touch the stars, longing to communicate 
with something we only imagine couid exist, ionging to know... Are 
we aione? 

Space expioration is a metaphor for iearning, and iearning is what 
drives our world. While realists continue to remind us of our current 
problems, dreamers stretch our minds. It is the dreamers' visions, hopes 
and desires that will lead us into the future. 



We ruin rain forests because there is oii under them, put mines in sacred 
ground for the sake of uranium. Wouid we also ruin another pianet for 
the sake of an answer to probiems of our own making? Of coursel 

Space expioration strengthens the dangerous beiief that human 
problems can be solved by our ever-increasing domination of the 
environment. Human beings wiil continue to feel at liberty to abuse 
naturai resources like rivers and rain forests if we know there is always 
another planet around the corner waiting to be expioited. 

We have done enough damage on Earth. We shouid ieave outer space 
aione. 



Ana 



Beatrice 



Dieter 
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The earth's resources are quickly dying out. The earth's population is 
increasing at a dramatic rate. Life cannot be sustained if we continue to 
live in such a way. Pollution has caused a hole in the ozone layer. Fertile 
lands are running out and soon our food resources will diminish. Already 
there are cases of famine and disease caused by over-population. 

Space is a vast empty region which we can use to our benefit. By 
supporting exploration into space, one day we may find a planet that 
we can live on. At the moment this seems unimaginable, but the notion 
of space travel was once thought of as impossible. Discontinuing space 
exploration in favour of solving immediate problems is a very narrow- 
minded and short-term view. We must learn to think not only for this 
generation but for the generations to come. 



Felix 



To ignore what the exploration of space has to offer would be a great 
loss to all mankind. The possibilities of gaining a greater understanding 
of the universe and its beginnings are too valuable to waste. The study 
of other celestial bodies has already increased our understanding of 
our environmental problems and the possible direction Earth could be 
heading in if we don't learn to manage our activities. 

There are also indirect benefits of research into space travel. The creation 
of laser technology and other medical treatments can be attributed to 
space research. Substances such as teflon have come out of mankind's 
quest to travel into space. Thus new technologies created for space 
research can have immediate benefits for everyone. 



Kate 



The stimulus for the unit Student Opinions consists of five short argumentative texts that offer opinions 
about space research. Because it is based on writing by students in their final year of school, the text is 
classified as educational in terms of situation. All of the short pieces that make up the stimulus for this unit 
are continuous, but as they were generated independently, and are juxtaposed for the purposes of the 
assessment, the format category is multiple texts. The stimulus is classified as argumentation, as the texts 
set forth propositions and attempt to persuade the reader to a point of view. 
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Question 1: STUDENT OPINIONS 

which of the following questions <\o the stu4ents seem to be responding toi’ 

A. Whgt is the rngjof problem facing the wofld tod^yi” 

B. Are you in f^vouf of sp^ce explofgtioni’ 

C. Do you believe in life beyond ouf pigneti’ 

D. Whgt recent gdvgnces hgve there been in spgce resegrchi’ 



Framework Characteristics 



Situation 


Educational 


Medium 


Print 


Text format 


Multiple 


Text type 


Argumentation 


Aspect 


Integrate and interpret: Form a broad understanding 


Question intent 


Identify a common theme across several short texts 


Item format 


Multiple choice 



CODING GUIDE 

Full Credit 

Code 1 : B. Are you in favour of space exploration? 

No Credit 

Code 0: Other responses. 

Code 9: Missing. 



Like many Forming a broad understanding items, this item requires the student to focus on more than 
one part of the text - deducing the theme by recognising a particular category of information implied 
in each of the five short texts. Students need to select a question from the four alternatives offered 
that applies to all five texts. Such a task indicates whether the student can distinguish key ideas and 
generalise. Less than two-thirds of the students gained credit for this item. Many of those who did not 
select the correct alternative opted for the first distractor, "What is the major problem facing the world 
today?" One reason for this may be that it plausibly applies to the first of the texts; and perhaps also it 
represents a kind of "stock issue" that students might expect to see, rather than the issue that all the texts 
are actually responding to. 
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Question 3; STUDENT OPINIONS 

which one of the wfitefs most 4i»'ectly contfgdicts Felix's gf-gumenti' 

A. Pieter. 

B. Ang. 

C. Kgte. 

P. Beatrice. 

Framework Characteristics 



Situation 


Educational 


Medium 


Print 


Text format 


Multiple 


Text type 


Argumentation 


Aspect 


Integrate and interpret; Develop an interpretation 


Question intent 


Recognising a counter-argument 


Item format 


Multiple choice 



CODING GUIDE 

Full Credit 

Code 1 : A. Dieter 

No Credit 

Code 0: Other responses. 

Code 9: Missing. 



The difficuity of this item - with less than one-third of the students in the field trial selecting the correct 
alternative - can be attributed to several factors. Firstly, it requires identifying a contrast, a skill that has 
been established empirically as Intrinsically more difficult than finding a point of similarity. Secondly, there 
are no explicit markers (such as "for" and "against") showing that contrary views are being expressed; the 
reader must infer the relationships. An added difficulty is that the comparisons and contrasts with Felix's 
opinion must be made several times, with each of the other four student opinions. Further, the basis of 
the contradiction is un-stated. Students need to generate their own categories for comparison, which 
in this case are not just opposing views about the merits of space research but also the grounds for the 
opinions; while both Ana and Dieter argue, in contrast to Felix, that space research is a bad idea, only 
Dieter directly contradicts Felix's view that the reason to pursue space research is to deal with the problem 
of diminishing resources on earth. 
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Question 6: STUDENT OPINIONS 

Thinking gbout the mgin idegs pfesented by the five students, which student do you ggfee with most 
stfonglyi' 

Student's ngme; 

Using youf own words, explain your choice by referring to your own opinion gn4 the mgin idegs 
presented by the student. 



Framework Characteristics 



Situation 


Educational 


Medium 


Print 


Text format 


Multiple 


Text type 


Argumentation 


/Aspect 


Reflect and evaluate: Reflect on and evaluate the content of a text 


Question intent 


/\ssess claims made in a text against personal knowledge and values 


Item format 


Open constructed response 



CODING GUIDE 

Full Credit 

Code I : Shows accurate comprehension of the chosen student's position regarding space exploration 

AND explains reason for agreeing with it. Must identify an argument or set of arguments 
which is unique to the chosen student (e.g. needs to do more than say whether the student is 
for or against space research). Must explicitly or implicitly refer to one of the main arguments 
of the chosen student, by: 

(1 ) introducing own supporting argument (in this case the answer may also quote or closely 
paraphrase the text); AND/OR 

(2) using own words to interpret or summarise the chosen student's argument. 

Summaries of each student's main argument(s) follow; 

Ana; Must state or imply that Ana is against space exploration and refer explicitly or implicitly to her 
argument that we should use resources to help humans (in need) rather than for space research. 

■ It is more important to help people on earth than to waste money on space technology. [Uses 
own words to rephrase AnaS argument.] 

■ Ana - 1 feel that we should take care of what is going on in our own world before we blow all 
our money on space exploration. I understand the importance of some exploration but I think 
disease and famine need to be helped out of this world first. [Summarises AnaS argument in own 
words and adds own comment.[ 
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Beatrice; Must state or imply that Beatrice is in favour of space exploration and refer 
explicitly or implicitly to her argument that space research is a positive expression of human 
endeavour. May refer to Beatrice's concern for taking a long-term view but must explicitly or 
implicitly distinguish her position from Felix's. 

■ Beatrice - "Space exploration is a metaphor for learning". I think it does not harm to widen our 
horizon. [Quotes from passage and adds own supporting argument in the second sentence.] 

Dieter: Must state or imply that Dieter is against space exploration and refer explicitly or 
implicitly to his argument that space exploration is connected with environmental damage OR 
that humans will damage space if they have a chance OR that exploring space encourages us 
to do more damage to Earth. Accept answers which suggest that Dieter's priority is to improve 
Earth's environment OR that we need to change ourselves or our attitudes. 

■ Dieter - 1 agree with him because he is concerned with the environment and he thinks that we 
should leave outer space alone. [Uses own words to summarise a main part of DieterS argument.] 

■ Dieter: Dieter says that we should stop wrecking the environment. I think this is the most 
important question facing our planet. [Summarises one of DieterS main points and adds 
supporting comment. Shows implicit understanding of DieterS position on space research. [ 

Felix: Must state or imply that Felix is in favour of space exploration and refer explicitly or 
implicitly to his argument that humans need to find another planet to live on AND/OR that life 
on earth cannot be sustained. May refer to Felix's concern about the environment but must 
explicitly or Implicitly distinguish his position from Dieter's. May refer to his concern that we take 
a long-term view but must explicitly or implicitly distinguish his position from Beatrice's. 

■ Felix: I agree with Felix because unless we are willing to face extinction, there is no other place to 
go after we have wrecked the earth. [Rephrases one of Felix^ main arguments.] 

Kate; Must state or imply that Kate is in favour of space exploration and refer explicitly or 
implicitly to her argument that space exploration leads to an increase in knowledge AND/OR 
we can apply what we learn from space exploration to other things. 

■ Kate: We are expanding our knowledge all the time in different ways because of space research. 
[Summarises Kate's main argument.] 

No Credit 

Code 0: Gives insufficient or vague answer or focuses on a (trivial) detail of the argument. 

■ /\na: We should not leave people behind as we attempt to forget and move on. [Does not go 
beyond close paraphrase.] 

■ /\na: I agree with Ana because the money spent on space research should be used to benefit 
those in need. [Does not go beyond close paraphrase.] 

■ Kate: Because she gives the best arguments. [Vague] 

■ Felix: Felix says that life cannot be sustained on earth, so we need to find another planet to live 
on. [Does not go beyond close paraphrase.[ 

■ Felix - the coolest. [Vague answer] 

■ /\na - her article is true and it is right. [Gives no account of main ideas of argument.[ 

■ Felix because the hole in the ozone layer is a serious problem. [Focuses on a detail of 
the argument.] 

■ Felix - he is really concerned about the environment. And he is a cool guy who respects nature. 
[Does not distinguish Felixk position from DieterK’ they are both concerned about the environment.] 

■ Beatrice because I agree that exploring space is important for our future. [Does not distinguish Beatrice^ 
position from FelixS: they are both in favour of space research and both refer to a long-term view] 
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OR: Shows inaccurate comprehension of the material or gives an implausible or Irrelevant answer. 

■ Beatrice: The human race will not be able to survive on planet earth much longer so we need to 
move somewhere else. [Inaccurate account of Beatrice's argument.] 

■ Kate - she is right because earth will soon run out of resources and then what will we do? 
[Inaccurate summary of Kate's argument (seems to confuse it with FelixS).] 

■ Challenge - 1 like challenges over all. [Irrelevant answer.] 

Code 9: Missing 



This task requires students to draw on their own knowledge and beliefs to evaluate the arguments put 
forward by the writers, comparing the substance rather than the form of the texts. In the five-aspect 
categorisation, this task is therefore classified as reflecting on the content of a text. In order to gain credit 
for this item, students needed to demonstrate implicitly or explicitly that they understood the main thrust 
of the argument advanced by their chosen writer, as well as justifying their position, either by introducing 
their own supporting argument or by summarising or interpreting the argument by the writer. 

Full credit was available regardless of which writer was nominated by the student, as long as the criteria 
outlined above were satisfied. Some typical answers that earned full credit were: 

^ Ana - 1 feel that we should take care of what Is going on in our own world before we blow all 
our money on space exploration. I understand the importance of some exploration but I think 
disease and famine need to be helped out of this world first. 

^ Dieter - 1 agree with him because he is concerned with the environment and he thinks that we 
should leave outer space alone. 

Felix - 1 agree with Felix because unless we are willing to face extinction, there is no other place to 
go after we have wrecked the earth. 

Answers that were not given credit often quoted from one of the texts without addition, whereas the task 
directed students to use their own words. Other responses that gained no credit were vague, or offered 
a general opinion without substantiating it with reference to one of the texts (for example, "Dieter. Let's 
look at the realities"). 



Question 7: STUDENT OPINIONS 

Some statements gre matters of opinion, bgse4 on the i4egs gn4 values oftbe writer. Some statements 
gre matters of fgct, which mgy be teste4 objectively gn4 3 re either correct or incorrect. 

Drgw g circle groun4 "matter of opinion" or "matter offset" next to egch of the quotations from the 
stu4ents' writing Iiste4 below. 

The first one hgs been 4one for you. 



Quotation from stu4ents' writing 


Matter of opinion or 
matter of fact? 


"Pollution has cause4 a hole in the ozone layer." CFelix) 


Matter of opinion /(Matter of fact 2 


"Billions of 4ollars are poure4 into space research by large 
companies each year." CAna) 


Matter of opinion / Matter of fact 


"Space exploration strengthens the 4angerous beliefthat 
human problems can be solve4 by our ever-increasing 
4omination oftbe environment." CDieter) 


Matter of opinion / Matter of fact 


"Discontinuing space exploration in favour of solving 
imme4iate problems is a very narrow-min4e4 an4 short- 
term view." CFelix) 


Matter of opinion / Matter of fact 
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Framework Characteristics 



Situation 


Educational 


Medium 


Print 


Text format 


Multiple 


Text type 


Argumentation 


Aspect 


Reflect and evaluate: Reflect on and evaluate the form of a text 


Question intent 


Distinguish fact from opinion 


Item format 


Complex multiple choice 



CODING GUIDE 



Quotation from stu4ents' weiting 


Matter of opinion or 
matter of fact? 


"Pollution hgs cgusecj g hole in the ozone Igyer." CFelix) 


C^^er of frcfP 


"Billions of 4ollgts gre poured into sp^ce research by Urge companies 
egch yegr." CAng) 


Matter of fact 


"Spgce exploration strengthens the dangerous beliefthgt humgn 
problems cgn be solved by our ever-increasing domination of the 


Matter of opinion 


environment." CPieter) 




"Piscontinuing spgce exploration in favour of solving immediate 
problems is g very ngrrow-minded 3nd short-term view." CFelix) 


Matter of opinion 



Full Credit 

Code 1 : 3 correct. 

No Credit 

Code 0: 2 or fewer correct. 

Code 9: Missing. 



The ability to distinguish between fact and opinion is an important critical faculty that individuals 
need to draw on every day In their capacity as citizens and consumers - the more so as the texts that 
we encounter in both the print and eiectronic media grow in number and sophistication. This task is 
designed to assess students' proficiency in this skill, but In a structured format that provides a good deai 
of support. Firstiy the challenge of the task is constrained by limiting the evaluation to short, clearly 
identified parts of the five texts. Two of the texts, moreover, contain terms that are clearly recognisable 
as markers of opinion: "dangerous belief" and "short-term view". Finaliy support for each fact/opinion 
decision is provided through the inclusion of definitions of "matters of opinion" and "matters of fact" in 
the stem of the question. With this scaffolding, just under three-quarters of the students in the field trial 
sample were able to categorise correctly all three of the quotations listed in the table. 
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Supermarket Notice 




/ 

/ 



Peanut Allergy Alert 

Lemon Cream Biscuits 

Date of alert: 04 February 
Manufacturer's Name; Fine Foods Ltd 
Product Information: 1 25g Lemon Cream 
Biscuits (Best before 1 8 June and Best before 
0 1 July) 

Details; Some biscuits in these batches may 
contain pieces of peanut, which are not included 
in the ingredient list. People with an allergy to 
peanuts should not eat these biscuits. 

Consumer action: If you have bought these 
biscuits you may return the product to the place 
of purchase for a full refund. 

Or call 1 800 034 24 1 for further information. 



/ 

/ 

/ 



This public notice consists of a very short text that has an everyday function: to warn about the possible 
danger of a product to consumers and to give advice to return the product for a refund. While the formatting 
of the stimulus reflects the International standard for product recall notices, many students may not have 
seen this kind of notice. Nevertheless, the content of the warning is clearly set out and a minimum number 
of words is used. Lemon biscuits were chosen as the product because of their familiarity and likely appeal. 

In developing very short easy items, the test developers sought to use simple pieces of stimulus with familiar 
content. This was not only to make the cognitive load of the items lighter, but also to present texts that were 
unlikely to Intimidate students with low reading proficiency, since such readers can easily be discouraged from 
even attempting to read something that they believe looks too hard or too long. The text format classification 
of the supermarket notice is non-continuous, as it consists of a list of described features. In terms of text type, 
the notice Is instructional; it provides directions on what to do if you have bought the product. 
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Question 2: SUPERMARKET NOTICE 

Whgt is the purpose of this noticei’ 

A. To gcjvertise Lemon Cregm Biscuits. 

B. To tell people when the biscuits were mg4e. 

C. To wgrn people gbout the biscuits. 

P. 7b expUin where to buy Lemon Cregm Biscuits. 

Framework Characteristics 




Situation 


Public 


Medium 


Print 


Text format 


Non-continuous 


Text type 


instruction 


Aspect 


integrate and interpret: Form a broad understanding 


Question intent 


Recognise the main idea of a short text by combining adjacent pieces 
of information 


Item format 


Multiple choice 



CODING GUIDE 

Full Credit 

Code I : C. To warn people about the biscuits. 

No Credit 

Code 0: Other responses. 

Code 9: Missing. 



To answer this question correctly, students must form a global understanding of the text to recognise its 
overall purpose. In particular, to reject distractors A and D, students must recognise that although the text 
is about a particular product, it is not an advertisement, but a warning. This item was easy. The easiness 
of this item comes in part from the fact that the whole text is very short. 



Question 3: SUPERMARKET NOTICE 

Whgt is the ngme of the company thgt mg4e the biscuitsi’ 



PISA 2009 ASSESSMENT FRAMEWORK - KEY COMPETENCIES IN READING, MATHEMATICS AND SCIENCE © OECD 2009 



195 




Situation 


Public 


Medium 


Print 


Text format 


Non-continuous 


Text type 


Instruction 


Aspect 


Access and retrieve: Retrieve information 


Question intent 


Locate a synonymous match in a short text 


Item format 


Closed constructed response 



CODING GUIDE 



Full Credit 

Code 1 : Fine Foods Ltd 

No Credit 

Code 0: Other responses. 

Code 9: Missing. 



To answer this question successfully the student needs to locate a single explicitly stated piece of 
information in the text, using a synonymous match between the task direction and the text 
(company / manufacturer). The fact that the whole text is very short, and that the needed information 
is near the beginning of the text, adds to the easiness of the task. The response format for the task is 
described as closed constructed response, since only one answer (with a small range of variants; Fine 
Foods or Fine Foods Ltd.) is given full credit. 



Question 5: SUPERMARKET NOTICE 

Whgt vvouM you <\o if you hg4 bought these biscuitsi’ 



why vvouM you 4o thiSi’ 

Vse information from the text to support your answer. 
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Framework Characteristics 



Situation 


Public 


Medium 


Print 


Text format 


Non-continuous 


Text type 


Instruction 


Aspect 


Reflect and evaluate: Reflect on and evaluate the content of a text 


Question intent 


Hypothesise about a personal course of action in response to the information in a text 


Item format 


Open constructed response 



CODING GUIDE 

Full Credit 

Code 1 : 5A: Provides a response that is consistent with an understanding that the biscuits may be 

returned with a refund. May refer to eating the biscuits, not eating the biscuits, returning them 
or getting rid of them in some other way AND 

5B: Gives an explanation consistent with the text and the response in 5A. Must be consistent 
with the idea that the peanuts pose a potential threat. 

■ (5A) 

Ask for my money back. 

(5B) 

It tells me to. 

I'm allergic to peanuts. 

They did something wrong. 

There might be something (else) wrong. 

I don't like peanuts. 

■ (5A) 

Throw them away. 

(5B) 

I'm allergic to peanuts. 

There might be something wrong. 

■ (5A) 

Eat them. 

(5B) 

Peanuts won't harm me . 

I'm not allergic to peanuts. 

I like peanuts. 

■ (5A) 

Give them to my classmate, 

■ (5B) 

She's not allergic to peanuts. 

■ (5A) 

Nothing. 

(5B) 

I'm not allergic to peanuts. 

I can't be bothered to go back to the shop. 
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5A: Quotes from or paraphrases an appropriate section of the text without further explanation 
(implying that the text tells you what to do and that no further explanation is required). 

5B: No response. 

■ (5A) Return the product to the place of purchase for a full refund. Or call 1 800 034 24 1 
for further information. 

(5B) (no response) 

■ (5A) Return the product to the place of purchase for a full refund. 

(5B) (no response) 

■ (5A) Call 1 800 034 24 1 for further information. 

(5B) (no response) 

■ (5A) Call the number for more information. 

(5B) (no response) 

5A: No response AND 5B: Gives explanation for taking no action. Must be consistent with the 
idea that the peanuts pose a potential threat. 

■ (5A) (no response) 

(5B) I'm not allergic to peanuts. 

■ (5A) (no response) 

(5B) I can't be bothered to go back to the shop. 

No Credit 

Code 0: Gives an insufficient or vague response. 

■ (5A) I don't know 

(5B) they might have peanuts 

■ (5A) eat them 

(5B) there might be peanuts 

Shows inaccurate comprehension of the material or gives an implausible or irrelevant response. 

■ (5A) (no response) 

(5B) check them for nuts. 

■ (5A) eat them. 

(5B) they look good enough to eat. 

■ (5A) give them to someone. 

(5B) it doesn't matter 

■ (5A) (no response) 

(5B) I'm allergic to peanuts. 

■ (5A) (no response) 

(5B) peanuts can be dangerous. 

■ (5A) throw them away. 

(5B) They're past their Best before date. 

Code 9: Missing. 

This question requires students to hypothesise about their likely personal response to the information in 
the text. Since the question requires a judgement based on personal preferences, or likely behaviours, the 
question is classified as reflect and evaluate. The coding guide indicates that a wide range of responses 
can receive full credit, so long as the response is consistent with two central ideas of the text; firstly, 
that it is possible to return the biscuits, and secondly that the biscuits pose a potential threat. The item is 
easy, with over four-fifths of the field trial respondents gaining full credit. The easiness of the item can be 
explained in part by the low level of reflection to be done: no specialised knowledge is required in order 
to explain a personal preference about a course of action regarding the familiar topic of food. 



PISA 2009 ASSESSMENT FRAMEWORK - KEY COMPETENCIES IN READING, MATHEMATICS AND SCIENCE © OECD 2009 



ANNEX A1 • PRINT READING SAMPLE TASKS 



Question 6: SUPERMARKET NOTICE 

why 4oes the notice inclu4e "Best before" 43tesi’ 



Framework Characteristics 



Situation 


Public 


Medium 


Print 


Text format 


Non-continuous 


Text type 


Instruction 


Aspect 


Integrate and interpret: Develop an interpretation 


Question Intent 


Identify the purpose of a conventional feature Included in a short text 


Item format 


Open constructed response 



CODING GUIDE 

Full Credit 

Code I : Refers to the fact that the Best before dates identify the batches of biscuits that are affected. 

■ to identify the batch(es). 

■ so you know which packets have peanuts. 

No Credit 

Code 0: Refers to when the biscuits should be eaten. 

■ because that's when you eat them. 

■ to tell you when to eat the biscuits. 

■ so you don't keep them too long. 

■ to tell you when they expire. 

Gives an insufficient or vague response. 

■ it's the date. 

Shows inaccurate comprehension of the material or gives an implausible or irrelevant 
response. 

■ so you know when the notice is irrelevant. 

Code 9: Missing. 



This question was answered correctly by less than one-third of students. Given the shortness and 
simplicity of the text, this illustrates the fact that the characteristics of a text only partly explain the 
difficulty of an item. The question requires students to identify the purpose of a specified part of the text, 
namely, the "best before dates". The difficulty of the item comes from the fact that students must focus on 
the purpose of the feature in this particuiar text. Students who answer by giving the usual purpose 
of this feature (that Is, to tell the consumer when the product should be used by) do not receive credit for 
this item. In this respect the full credit response is contrary to expectations, an established marker of item 
difficulty. 
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PRINT READING UNIT 5 



The Motorcycle 



Have you ever woken up feeling that something was wrong? 
It was a day like that for me. 

I sat up in bed. 

A little later I opened the curtains. 

It was terrible weather - the rain was pouring down. 

Then I looked down into the yard. 

YesI There it was - the motorcycle. 

It was just as wrecked as last night. 

And my leg was starting to hurt. 



Like the Supermarket Notice, this short, self-contained story was included in order to target students with iow 
reading proficiency. Both the topic and the shortness of this text contribute to its likely appeal. The items that 
reiate to this text focus on the interpretation both of specific parts of the text, and of the overail action of the 
text (a motorcycle accident and its aftermath), which although strongly implied, is never directly stated. As 
with all narrative texts, this piece is classified as personai in terms of situation, and as continuous in text format. 
All items relating to this text functioned well in the field trial. The text was not chosen for inclusion in the main 
survey because the narrative text type was aiready well covered by other pieces. 



Question 1 : THE MOTORCYCLE 

Something hg4 happened to the person in the story the night before. Whgt wgs iti’ 

A. The bg4 wegther hg4 43mgge4 the motorcycle. 

B. The bg4 weather hg4 stoppe4 the person from going outsi4e. 

C. The person hg4 bought g new motorcycle. 

P. The person hg4 been in g motorcycle gcci4ent. 

Framework Characteristics 



Situation 


Personal 


Medium 


Print 


Text format 


Continuous 


Text type 


Narration 


Aspect 


Integrate and interpret: Develop an interpretation 


Question intent 


Infer prior events using information in a short story 


Item format 


Multiple choice 
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CODING GUIDE 

Full Credit 

Code I : D. The person had been in a motorcycle accident. 

No Credit 

Code 0: Other responses. 

Code 9: Missing. 




Ail three items relating to this text were easy, and this item was the easiest. The item requires students 
to make an inference about events that happened before the time of the story by using the information 
that is given in the story. The item stem gives a general statement: students are not directed to the part 
of the story containing the critical information (the final two lines). However, the fact that the whole text 
is very short makes it easier for students to locate the information reievant to the task. Having located the 
relevant information, matching the words "motorcycle", "wrecked" and "hurt" in the text to "motorcycle 
accident" in the response, is a simple inference. 



Question 2: THE MOTORCYCLE 

"It W 3 S g like th^t for me." 

Whgt kind of 4gy did this turn out to bei’ 

A. A good d^y. 

B. A b^d dgy. 

C. An exciting d^y. 

D. A boring d^y- 

Framework Characteristics 



Situation 


Personal 


Medium 


Print 


Text format 


Continuous 


Text type 


Narration 


Aspect 


Integrate and interpret: Develop an interpretation 


Question intent 


Relate a descriptive phrase to events implied in a very short story 


Item format 


Multiple choice 



CODING GUIDE 

Full Credit 

Code I : B. A bad day. 



PISA 2009 ASSESSMENT FRAMEWORK - KEY COMPETENCIES IN READING, MATHEMATICS AND SCIENCE © OECD 2009 



201 



ANNEX A1 • PRINT READING SAMPLE TASKS 



No Credit 

Code 0: Other responses. 

Code 9: Missing. 



This item was very easy. The item requires students to use their understanding of the whole story to 
interpret the descriptive phrase "It was a day like that for me". Students who receive full credit for the item 
recognise that the event implicitly referred to in the story (a motorcycle accident) suggests a bad day. The 
three incorrect distractors were chosen with about equal frequency. Again, the overali shortness of the 
text contributes to the easiness of this item, making it accessibie even to students of iow proficiency. 



Question 6: THE MOTORCYCLE 

why does the writer begin the story with g questioni’ 

A. Because the writer wgnts to know the answer. 

B. 7b involve the reader in the story. 

C. Because the question is hgrd to answer. 

P. To remind the reader th^t this kind of experience is rgre. 

Framework Characteristics 



Situation 


Personal 


Medium 


Print 


Text format 


Continuous 


Text type 


Narration 


Aspect 


Reflect and evaluate; Reflect on and evaluate the form of a text 


Question intent 


Recognise the purpose of beginning a story with a rhetorical question 


Item format 


Multiple choice 



CODING GUIDE 



Full Credit 

Code I : B. To invoive the reader in the story. 

No Credit 

Code 0: Other responses. 

Code 9: Missing. 



The final item reiating to this text was the hardest item in the set, but it was still easy, with over four-fifths of the 
triai test participants gaining fuil credit. The item requires students to reflect on the form of the text by recognising 
the author^ purpose in beginning the story with a rhetorical question. To answer this question correctly, students 
must interpret the rhetorical question both as a literary device and in terms of its specific content. Distractors A 
and C outline two common purposes of rhetorical questions that are not the purpose of the device in this text. 
Distractor D relates the rhetorical question to the content of the text, but suggests a position that is opposite to 
that implied in the text: the story suggests that the kind of experience the writer is having is not rare, but common. 
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PRINT READING UNIT 6 



Tall Buildings 

"Tall buildings" is an article from a Norwegian magazine published in 2006. 




Figure I : Tall buildings of the world 

Figure I shows the number of buildings of at least 30 
storeys that have been built, or are under construction 
This includes buildings that have been proposed since 
January 200 1 . 
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Figure 2: Some of the world's tallest 
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This text juxtaposes two figures that are looseiy related in terms of their content. Both figures are about 
tall buildings in the world: Figure I shows the number of tall buildings in various cities, proposed or 
already built, while Figure 2 shows some of the world's tallest buildings. Both Figure I and Figure 2 
are graphs. Although each figure is introduced by a small piece of expianatory prose, the substantive 
information of this text is given in the two figures, making the overail text format non-continuous. The 
text type is description, while the situation of this text is educational since it appeared in a magazine 
for students. The piece begins with a brief introduction expiaining its context, both in terms of time (the 
piece was published in 2006) and place (the magazine is Norwegian). One of the reasons why this unit 
was not chosen for inclusion in the main survey was because of concerns about the ephemeral nature of 
the material: as more and more tall buildings are built in the world, the material will very quickly become 
out-dated. 
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Question 1: TALL BUILDINGS 



when the mgggzine article wgs published, which of the buildings in Figure 2 wgs the tgllest completed 
buildingi’ 



Framework Characteristics 



Situation 


Educational 


Medium 


Print 


Text format 


Non-continuous 


Text type 


Exposition 


Aspect 


Access and retrieve; Retrieve information 


Question intent 


Locate information that is explicitly stated in a graph 


Item format 


Closed constructed response 



CODING GUIDE 

Full Credit 

Code 1 : The CN Tower. 

No Credit 

Code 0: Other responses. 

■ The Burj Tower. 
Code 9: Missing. 



This item was answered correctiy by about two-thirds of students. The difficulty in this item comes from 
the need to integrate information from more than one place in the text. To receive credit, students must 
connect the information in the question ("when the magazine article was published" and "compieted 
building") with the introduction to Figure 1 . This introduction explains that, at the time of the articie's 
publication, the Burj Tower was not yet complete, and so eliminates this apparently plausible option as a 
correct answer. Students could also arrive at the correct answer by relating the date in the introduction 
to the unit (Tail buildings" is an articie from a Norwegian magazine published in 2006) to the date given 
for "Burj Tower" in Figure 2 (2008) to eiiminate this as a possibie answer. Students who read only "tallest 
building" in the item stem, and answer by referring to the tallest building in Figure 2 (the Burj Tower), do 
not receive credit for this item. 



Question 4: TALL BUILDINGS 

Wbgt kind of information does Figure 1 providei' 

A. A comparison of the heights ofdifferent buildings. 

B. The totgl number of buildings in different cities. 

C. The number of buildings gbove g certain height in various cities. 
P. Information gbout styles of buildings in different cities. 
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Framework Characteristics 



Situation 


Educational 


Medium 


Print 


Text format 


Non-continuous 


Text type 


Exposition 


Aspect 


Integrate and interpret; Form a broad understanding 


Question intent 


Generalise about the type of information presented in a graph 


Item format 


Multiple choice 



CODING GUIDE 

Full Credit 

Code I : C. The number of buildings above a certain height in various cities. 

No Credit 

Code 0: Other responses. 

Code 9: Missing. 



Like the first item in this unit, this item was moderately difficult. It was answered correctly by about 
two-thirds of students. While the question stem specifically directs students to Figure 1 , and the 
introduction to Figure 1 is a synonymous match with the key, C, about half of those who did not gain 
credit for the item selected alternative A, which relates to Figure 2. 



Question 5: TALL BUILDINGS 

The Rg4isson SAS in Oslo, Norway is only 117 metres tgll. Why h^s it been inclu4e4 in Figure 2 ? 



Framework Characteristics 



Situation 


Educational 


Medium 


Print 


Text format 


Non-continuous 


Text type 


Exposition 


Aspect 


Reflect and evaluate; Reflect on and evaluate the content of a text 


Question intent 


Recognise the influence of reader's perspective on the way a text is constructed 


Item format 


Multiple choice 
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CODING GUIDE 

Full Credit 

Code I : Refers to the fact that the article is from a Norwegian magazine or that the readers are likely to 

be Norwegian. 

■ It's from a Norwegian magazine. 

■ It's written for people in Norway, so its to give them a sense of perspective. 

■ To show the people in Norway that they really don't have very tall buildings! 

Refers to the SAS Plaza being a point of reference or comparison without referring to 
Norwegian readers. 

■ For comparison. 

■ To have a reference point. 

■ So you get an understanding of what the scale means. 

No Credit 

Code 0: Gives an insufficient or vague response. 

■ I 1 7 metres is still pretty tall. 

■ Because It is one of the tallest buildings in the world. 

■ It's 30 or more storeys high. 

■ It's the tallest building in Norway, [too vague - does not link with the audience of the text] 

■ Because it's Norwegian, [too vague - does not link with the audience of the text] 

Shows inaccurate comprehension of the material or gives an implausible or irrelevant 
response. 

■ It's the only one that's a hotel, [true, but not why it is included in this text] 

■ It's the only one without a spire, [true, but not why it is included in this text] 

Code 9: Missing. 



This item was difficult. Only about one-quarter of students answered correctly. The coding guide outlines 
two kinds of answers that would each receive full credit. In the Field Trial, these two kinds of answers 
were given different codes, but the field trial data showed that there was little difference between the 
abilities of the groups of students who answered in these two different ways. Therefore if it had been 
included in the main survey selection, this item would have been coded with a single full credit code as 
shown here. Both kinds of full credit answer require student to recognise that the purpose for including 
a specified building in Figure 2 is to give a point of reference to the reader. In the first kind of full credit 
answer, students do this by relating the information in the introduction (that the article was published in 
a Norwegian magazine) to the author's decision to include the tallest building in Norway (the Radisson 
SAS Plaza) in Figure 2. In the second kind of full credit answer, the idea of perspective is expressed in 
general, rather than specific terms. 
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Question 9; TALL BUILDINGS 

Suppose thgt infofmgtion gbout tgll bui'Mings wgs pfesented 333 in in gn article like this in twenty 
yegfs' time. 




Listed below gre two features of the original article. Show wbetbei" or not these features gre likely to 
cbgnge in twenty yegrs' time, by circling either "Yes" or "No" in the tgble below. 



Feature of Article 


Is it likely to change in twenty years? 


The title of Figure 2. 


Yes / No 


The numbers of buiMings shown in Figure 1. 


Yes / No 



Framework Characteristics 



Situation 


Educational 


Medium 


Print 


Text format 


Non-continuous 


Text type 


Exposition 


Aspect 


Reflect and evaluate; Reflect on and evaluate the content of a text 


Question intent 


Distinguish between structural features and content of a set of graphs 


Item format 


Complex multiple choice 



CODING GUIDE 

Full Credit 

Code I : Both correct; No, Yes in that order. 

No Credit 

Code 0: Other responses. 

Code 9: Missing. 



In this complex multiple-choice item, students are required to make a decision about two elements In 
the text, in each case choosing the correct response from two options. Students receive full credit only 
if the correct answer is identified for both elements. The item was easy, with nearly three-quarters of 
students receiving full credit. This item requires students to draw on their own knowledge and apply It 
to the text, and is therefore classified as reflect and evaluate. Students must decide whether each of the 
features listed is stable, or whether it would change over time. The first item in the list refers to part of the 
conceptual structure of the text. The second is about ephemeral content, the record of a point in time. 
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Democracy in Athens 

PART A 

Thucydides was a historian and military man who lived in the fifth century BC, during the Classical 
Greek period. He was born in Athens. During the Peloponnesian War (43 1 BC to 404 BC) between 
Athens and Sparta he was in command of a fleet whose mission was to protect the city of Amphipolis 
in Thrace. He failed to reach the city in time. It fell into the hands of Brasidas, the Spartan general, 
which forced Thucydides into a twenty-year exile. This granted him the opportunity of collecting 
detailed information from the two warring factions and the possibility of doing research for his work 
History of the Peloponnesian War. 

Thucydides is regarded as one of the great historians of Ancient times. He focuses on natural causes 
and the behaviour of each individual rather than on fate or the intervention of divinities to explain the 
evolution of History. In his work, facts are not presented as mere anecdotes; rather, they are explained 
in an attempt to find out the reasons that led the main characters to act as they did. Thucydides' 
emphasis on the behaviour of individuals is why he sometimes introduces fictitious speeches: these 
help him explain the motivations of the historical characters. 



Thucydides attributes to Pericles (fifth century BC), the Athenian ruler, the following speech in honour of 
the soldiers who fell in the first year of the Peloponnesian War. 

Our system of government does not copy the laws of neighbouring states; we are rather a pattern to 
others than imitators ourselves. Our system is called democracy, since its administration depends on 
the many instead of the few. Our laws afford equal rights to all in their private affairs, whereas the 
prestige in public life depends on merit rather than on social class. 

Social class does not prevent a person from holding any public position either (...). And, at the same 
time that we do not interfere in private affairs, we do not break the law as regards public matters. 

We give our obedience to those whom we put in positions of authority, and we obey the laws 
themselves, especially those which are for the protection of the oppressed, and those unwritten laws 
which it is an acknowledged shame to break. 

Furthermore, we provide plenty of means for the pleasure of the mind. The games and sacrifices we 
celebrate all the year round, and the elegance of our private places of residence, form a daily source 
of pleasure that helps to banish any worry; while the many inhabitants of the city draw the produce 
of the world into Athens, so that to the Athenian the fruits of other countries are as familiar as those 
of his own. 

Democracy in Athens consists of two fairly short but dense texts. The first is classified as expository, 
although the first paragraph if considered alone could better be described as narration, since it gives an 
account of when something happened, referring to a sequence of events in a person's life. However, 
in the context of the whole of Part A, the biographical paragraph serves as an introduction to the more 
substantial second paragraph, which places Thucydides in the context of ideas, describing his originality 
as an historian. Part A as a whole, then, provides an explanation of concepts or mental constructs, which 
is a marker of expository texts. 

Part B presents a sample of one of the "fictitious speeches" written by Thucydides that are referred to in 
Part A. Part B is an argumentative text, words imagined as having been spoken by Pericles in a speech 
of political persuasion. Part of the challenge of reading the stimulus as a whole is understanding the 
relationship between the two texts: it is not stated explicitly but can be inferred from the last sentence of 
Part A and the introduction to Part B. Other features that make this stimulus likely to be relatively difficult 
for 1 5-year-olds are its remoteness from their everyday experience, the abstractness of the language and 
the formal register, particularly of the rendition of Thucydides' writing. On the other hand it is reasonable 
to suppose that most students at the end of their compulsory schooling are fairly familiar with history 
as a concept, and some notion - even if not necessarily articulated - of what democracy might be. This 
assumed background was judged to provide sufficient context for students to approach the Democracy 
in Athens material. 



PART B 
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Question 3: DEMOCRACY IN ATHENS 

One pufpose of the speech in Pgrt B wgs to honouf soMiefs who fell in the fifst yegf ofthe 
Peloponnesian Wgn 

Whgt wgs ANOTHER purpose ofthis speech i* 



Framework Characteristics 



Situation 


Educational 


Medium 


Print 


Text format 


Multiple 


Text type 


Exposition 


Aspect 


integrate and interpret; Form a broad understanding 


Question intent 


identify the purpose of an argumentative text by linking it to its context 


Item format 


Open constructed response 



CODING GUIDE 

Full Credit 

Code 2: Refers to (political) motivation attributed to Pericles. May refer to: persuading soldiers to 

continue the struggle; consoling the families of the dead; fostering pride in the citizens of 
Athens; or stressing the virtues of Athens compared to Sparta or other cities. Response must be 
plausible and consistent with the text. 

■ To make people proud of Athens. 

■ To promote democracy. 

■ To explain benefits of the Athenian democracy. 

■ Making people think Athens is still ok, despite the fact that they are in trouble right now. 

■ To reinforce positive thinking and positive attitudes. 

■ To rouse the people. 

■ To promote patriotism. 

■ To win the next election. 

■ To get more popular. 

■ To make people aggressive against Spartans. 

Refers to Thucydides' purpose of understanding Pericles' motivation or way of thinking. 

■ To explain Pericles' motivation/psychology. 

■ To explain why he did what he did. 

Partial Credit 

Code I : Response refers only to explaining how democracy works. 

■ To introduce democracy. 

■ To explain democracy to the people. 

No Credit 

Code 0: Gives an insufficient or vague response. 
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■ To honour the dead soldiers. [Repeats stem.] 

Shows inaccurate comprehension of the material or gives an implausible or Irrelevant 
response. 

■ Athens was the subject of the speech, [purpose not mentioned] 

■ To make people laugh, [inaccurate] 

Code 9: Missing. 



Question 4: DEMOCRACY IN ATHENS 

why WTS Tbucy4i4es fotce4 into exilei’ 

A. He WTS unable to achieve victory for the Athenians gt Amphipolis. 

B. He took over g fleet in Amphipolis. 

C. He collecte4 information from the two wgrring factions. 

P. He 4eserte4 the Athenians to fight with the SpTrtgns. 

Framework Characteristics 



Situation 


Educational 


Medium 


Print 


Text format 


Continuous 


Text type 


Exposition 


Aspect 


Access and retrieve: Retrieve information 


Question intent 


Locate information that is explicitly stated In a dense text 


Item format 


Multiple choice 



CODING GUIDE 



Full Credit 

Code I : A. He was unable to achieve victory for the Athenians at Amphipolis. 

No Credit 

Code 0: Other responses. 

Code 9: Missing. 



This item was moderately difficult. About two-thirds of students answered correctly. While the information 
required is explicitly stated, the match between the key "He was unable to achieve victory for the 
Athenians at Amphipolis" and the relevant part of the text is far from synonymous, and students must 
make connections within the text to retrieve it. To answer correctly, students need to understand both 
Thucydides' position within the Peloponnesian War (he was Athenian) and his mission during that conflict 
(to protect the city of Amphipolis). They must then establish the referent for "it" (It fell Into the hands of 
Brasidas, the Spartan general, which forced Thucydides into a twenty-year exile) to Identify the causal 
relationship between the events in the text. Another source of difficulty in this item lies in the fact that it 
relates to a dense text, containing complex ideas that are likely to be unfamiliar. 
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This is coded as a partial credit item. To gain full credit, the response needs to identify the purpose 
of Pericles' speech by linking it to its context, which is partly provided in the expository text as well 
as more directly by the speech itself and by its introduction. The full credit response thus needs to be 
underpinned by a full and detailed understanding of both texts, and to use a high level of inference in 
an unfamiliar context, dealing with some ambiguity (about the authorship of the speech). Responses in 
this category refer to the political motivation attributed to Pericles, possibly by mentioning such purposes 
as persuading soldiers to continue the struggle; consoling the families of the dead; fostering pride in the 
citizens of Athens; or stressing the virtues of Athens compared to Sparta or other cities. Alternatively full 
credit answers could refer to the more abstract level of meaning implied in Part A: Thucydides' purpose 
of understanding Pericles' psychology or way of thinking. Full-credit answers do not need to be long or 
elaborate yet just over a quarter of responses in the field trial were in this category. About one-fifth of 
participants in the field trial were able to demonstrate a partial understanding of the text, indicating that 
its substantial content was about democracy but without registering the persuasive intent. 






Question 5: DEMOCRACY IN ATHENS 

who wrote the speech in Pgrt Bi’ Refer to the text to support your answer. 



Framework Characteristics 



Situation 


Educational 


Medium 


Print 


Text format 


Multiple 


Text type 


Exposition 


Aspect 


Integrate and interpret; Develop an interpretation 


Question intent 


Recognise the relationship between two texts by integrating information 
provided 


Item format 


Open constructed response 



CODING GUIDE 

Full Credit 

Code I : Identifies Thucydides (explicitly or implicitly) as the author of the speech AND refers to 

Thucydides attributing the speech to Pericles. May paraphrase or quote the text directly. 

■ Thucydides. It says 'Thucydides attributes to Pericles". 

■ Thucydides. "He sometimes introduces fictitious speeches: these help him explain the motivations 
of the historical characters." 

■ It tells you that Thucydides made up speeches for the people he was writing about. 

No credit 

Code 0: Identifies Thucydides as the author of the speech without explanation. 

■ Thucydides. 

■ A historian and military man. 

Gives an insufficient or vague response. 
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■ Somebody else, [too vague] 

Shows inaccurate comprehension of the material or gives an implausible or irrelevant response. 

■ Pericles. 

■ He was writing in the Peloponnesian War. 

Code 9: Missing. 



This item was very difficult. It was answered correctly by just under one-guarter of students. The answer 
is unambiguous when one integrates information from the two texts (Part A and Part B), so the text 
format for this item is multiple. The juxtaposition of the two texts makes it clear that the author of 
Pericles' speech is Thucydides. This may be deduced by linking the final part of Part A (which explains 
that Thucydides sometimes introduces fictitious speeches in his attempts to explain the motivations of 
historical characters) with the introduction to Part B (which begins "Thucydides attributes to Pericles..."). 
Alternatively, students may focus only on the introduction to Part B, inferring that "attributed" implies 
an invention (on Thucydides' part). An added difficulty of this task is that to receive credit, students 
must support their answer by referring to the text. The reference to the text may be in the form of direct 
quotation, paraphrase or generalisation. 



Question 6: DEMOCRACY IN ATHENS 

Accor4ing to the text, whgt mg4e Thucy4i4es 4ifferent from the other historians of his time] 

A. He wrote gbout every4gy people, not heroes. 

B. He use4 gnec4otes rgther thgn mere f^cts. 

C. He explgine4 historical events by referring to their supernatural cguses. 

D. He focuse4 on whgt mg4e people get the wgy they 4i4- 

Framework Characteristics 



Situation 


Educational 


Medium 


Print 


Text format 


Continuous 


Text type 


Exposition 


Aspect 


Access and retrieve: Retrieve information 


Question intent 


Locate information that is explicitly stated in a dense text 


Item format 


Multiple choice 



CODING GUIDE 

Full Credit 

Code I : D. He focused on what made people act the way they did. 

No Credit 

Code 0: Other responses. 

Code 9: Missing. 
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This item was difficult. It was answered correctly by less than one-half of students. While the information 
required is explicitly stated, it is located in a dense text. A further source of difficulty is that the question does 
not explicitly direct students to the part of the text containing the answer. Students who answer correctly 
must locate the correct part of the text by matching "what made Thucydides different" in the stem with "He 
focuses on. ..rather than..." in the stimulus. When the correct part of the text is located, students can match 
"what made people act the way they did" in the item to any one of "behaviour of each individual", "reasons 
that led the main characters to act as they did" and "motivations of the historical characters" in the text. 



Question 7: DEMOCRACY IN ATHENS 

Look gt this pgrt of the text, from negr the end of Pgrt B; 

"Furthermore, we provide plenty of megns for the pleasure of the mind- The ggmes gnd sacrifices we 
celebrate gll the yegr round, 9nd the eleggnce of out private places of residence, form g d^ily source of 
pleasure thgt helps to bgnish gny worry." 

which of these sentences best summarises this pgrt of the texti" 

A. The system of government in Athens gllows gny person to create Igws. 

B. Entertainment gnd beguty gre pgrt of the good life you cgn hgve in Athens. 

C. Athenians live in too much luxury gnd cgnnot tgke life seriously. 

D. Public gnd private life gre seen gs the sgme thing. 

Framework Characteristics 



Situation 


Educational 


Medium 


Print 


Text format 


Continuous 


Text type 


Argumentation 


Aspect 


Integrate and interpret; Develop an interpretation 


Question intent 


Construe a sentence from an argumentative text 


Item format 


Multiple choice 



CODING GUIDE 



Full Credit 

Code I : B. Entertainment and beauty are part of the good life you can have in Athens. 

No Credit 

Code 0: Other responses. 

Code 9: Missing. 



This item was moderately difficult. It was answered correctly by less than two-thirds of students. In this item, 
students are required to interpret a part of the text that is quoted in full in the question stem. The difficulty in 
this item therefore lies purely in construing the text, since no retrieval of information is required. The part of the 
text to be interpreted is fairly long and complex, and contains several ideas. Students who answer correctly 
must negotiate multiple inferences, matching "elegance of our private places" in the text with "beauty" in the 
stem, as well as "pleasure of the mind" and "games" in the text with "entertainment" in the question. 
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Destination Buenos Aires 

And so the three mail planes from Patagonia,' Chiie and Paraguay were returning from the South, 
the West and the North to Buenos Aires. Their cargo was awaited there so that the plane for Europe 
could take off, around midnight. 

Three pilots, each behind an engine casing heavy as a barge, lost In the night, were contemplating 
their flight and, approaching the immense city, would descend slowly out of their stormy or calm sky, 
like strange peasants descending from their mountain. 

Riviere, who was responsible for the entire operation, was pacing up and down on the Buenos Aires 
landing-ground. He remained silent, for until the three planes had arrived, the day heid a sense of 
foreboding for him. Minute by minute, as the telegrams reached him. Riviere was conscious that he 
was snatching something from fate, gradually reducing the unknown, hauling in his crews out of the 
night, towards the shore. 

One of the men came up to Riviere to give him a radioed message: 

Chile mail reports that he can see the lights of Buenos Aires. 



Before long. Riviere would hear this plane; already the night was surrendering one of them, just as a 
sea, swollen with ebbing and flowing and mysteries, surrenders to the shore the treasure It has tossed 
around for such a long time. And later on, it would give back the other two. 

Then this day's work would be over. Then the worn-out crews would go and sleep, to be replaced 
by fresh crews. But Riviere would have no rest; the mail from Europe, In Its turn, would fill him with 
apprehension. And so it would always be. Always. 



Destination Buenos Aires is an extract from Antoine de Saint-Exupery's 1 93 1 novel Vo! de Nuit (published 
in English as Night Fiight). The only addition to the original text for its appearance in PISA was an 
explanatory footnote relating to "Patagonia", as students would certainly have differing levels of 
familiarity with this place name. The explanation gives context which might help students to negotiate 
the text. The extract takes place at a landing-ground in Buenos Aires, and is a self-contained portrait of 
Riviere, a man weighed down by the responsibility of his job. Though the novel was written in 1 93 1 , the 
human themes remain familiar. 



Question 3: DESTINATION BUENOS AIRES 

How does Riviere feel gbout bis jobi’ Hse the text to give g regson to support your answer. 



Good. 



Antoine de Saint-Exupery Vo! de Nuit, © Editions Gallimard 
1 . Southern region of Chile and Argentina 
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Framework Characteristics 



Situation 


Personal 


Medium 


Print 


Text format 


Continuous 


Text type 


Narration 


Aspect 


Integrate and interpret; Develop an interpretation 


Question intent 


Link information across a narrative to generalise about a character's state of 
mind, providing evidence to support the generalisation 


Item format 


Open constructed response 



CODING GUIDE 

Full Credit 

Code 2: Describes Riviere's feeling about his job by referring to stress, persistence, being burdened, or 

being committed to doing his duty; AND gives an explanation referring to a relevant section 
of the text. May refer to the text generally, or may paraphrase or quote the text directly. The 
quotation must match the stated emotion. 

■ He is overwhelmed by it all, you can see In the last line, he never gets to rest. 

■ He is stressed. The day has "held a sense of foreboding for him". 

■ He is weighed down by it. All day he worries about those three planes, then he has to worry 
about the Europe onel 

■ He is resigned. You can see from that last "always" that he thinks things will never change. 

■ He really cares about his job. He can't relax until he knows that everyone is safe. [Includes a 
general reference to the text.] 

Partial Credit 

Code I : Describes Riviere's feeling about his job by referring to stress, persistence, being burdened, or 

being committed to doing his duty, without an explanation that refers to the text. 

■ He feels really responsible for the things that happen. 

■ HeS stressed. 



No Credit 

Code 0: Gives an insufficient or vague response. 

Shows inaccurate comprehension of the material or gives an implausible or irrelevant response. 

■ He likes his job because he is in control of lots of things, [not supported by the text] 

■ He thinks it is cool because he can watch planes, [not supported by the text] 

Code 9: Missing. 



The coding guide for this item shows that there are two kinds of response that receive credit. Full 
credit responses are those which accurately respond to the question and give an explanation using 
the text. Partial credit responses are those which accurately respond to the question, but fail to give an 
explanation for the response. The partial credit code recognises that an incomplete answer is superior 
to an inaccurate one. In the field trial, less than half of the students received full credit for this item, but 
an additional one quarter received partial credit, meaning that about three-quarters of students received 
some credit (either full or partial] for this item. This question is classified as integrate and interpret. 
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because although students are required to generate a response that is not given explicitly in the text, all 
the information necessary to answer the question is contained within the text. 



Question 5: DESTINATION BUENOS AIRES 

"Pestingtion Buenos Aires" wgs written in 1931 Po you think th^t nowgd^ys Riviere's concerns woul4 
be simiUri’ Give 3 reason for your answer. 



Framework Characteristics 



Situation 


Personal 


Medium 


Print 


Text format 


Continuous 


Text type 


Narration 


Aspect 


Reflect and evaluate; Reflect on and evaluate the content of a text 


Question intent 


Hypothesise about the effect on a character of a change in a narrative's context 


Item format 


Open constructed response 



CODING GUIDE 

Full Credit 

Code 1 : Answers (or implies) Yes OR No and refers to a time-based comparison AND supports their 

answer. May refer to material concerns such as technological progress or improvements in 
security OR to psychologicai concerns such as anxiety. Answer must be consistent with an 
accurate reading of the text. 

■ Now, pilots (planes) have very sophisticated tools intended for orientation, making up for technical 
issue when the weather conditions are bad. 

■ No, nowadays, planes have radars and automatic piloting systems, which can help them to 
escape from dangerous situations. 

■ Yes, planes are still dangerous, just like any other means of transport. The risks of crash or engine 
failure are never eradicated. 

■ Now, new technologies and technical progress are very important, in the planes as well as on the 
ground. 

■ Yes, there is still a risk of crashing. 

■ No, before, there was no fear of terrorist attacks. 

No Credit 

Code 0: Gives an insufficient or vague response. 

■ No, the fears are different today. 

■ Yes, some progress has been made. 

■ In a way, yes, but in the modern day context, [vague] 

■ Over the years, people would have changed it. [vague] 
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Shows inaccurate comprehension of the material or gives an implausible or Irrelevant response. 

■ No, because you don't travel by night nowadays, [inaccurate about the world] 

■ No, because nowadays, pilots are much better trained, [irrelevant] 

■ No, Riviere is really happy with his job but nowadays there are terrorists to worry about. 

[inaccurate reading of the text] 






Code 9: Missing. 

This item was moderately difficult. Just over one-half of students answered correctly. The Item requires 
students to reflect on the context in which a text was written and compare that context to their own. The 
object of the question is to encourage reflection. Therefore, so long as the response Is consistent with an 
accurate reading of the text, and expresses a plausible position about the modern day context, a wide 
range of responses receive full credit, regardless of the position adopted. 



Question 7: DESTINATION BUENOS AIRES 

Whgt happens to the mgin cbgrgcter in this text] 

A. He h^s gn unplegsgnt surprise. 

B. He 4ed4es to change his job. 

C. He wgits for something to happen. 

D. He legrns to listen to others. 

Framework Characteristics 



Situation 


Personal 


Medium 


Print 


Text format 


Continuous 


Text type 


Narration 


Aspect 


Integrate and interpret; Form a broad understanding 


Question intent 


Recognise the main action in a narrative text 


Item format 


Multiple choice 



CODING GUIDE 

Full Credit 

Code I : C. He waits for something to happen. 

No Credit 

Code 0: Other responses. 

Code 9: Missing. 
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This item was easy. About three-quarters of students answered correctly. The item requires students to 
demonstrate a broad understanding of the text by identifying its main idea. The item requires making 
links across the text and generalising about its overall action. The easiness of the item comes from the fact 
that the main idea of the text is implied and reinforced across the whole text. 



Question 8: DESTINATION BUENOS AIRES 

Accor4ing to the second Ust pgrggrgph ("Before long in whgt wgy gre the night gnd 3 seg simiUti’ 

A. Both hide whgt is in them. 

B. Both gre noisy. 

C. Both hgve been tgmed by humgns. 

P. Both gre dangerous to humans. 

E. Both gre silent. 

Framework Characteristics 



Situation 


Personal 


Medium 


Print 


Text format 


Continuous 


Text type 


Narration 


Aspect 


Integrate and interpret; Develop an interpretation 


Question intent 


Understand the point of comparison in a metaphor 


Item format 


Multiple choice 



CODING GUIDE 

Full Credit 

Code I : A. Both hide what is in them. 

No Credit 

Code 0: Other responses. 

Code 9: Missing. 
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The item requires students to interpret a metaphor, aithough the word "metaphor" is deliberate^ avoided 
in the stem: such metalinguistic terms are likely to vary in familiarity for students from different educational 
backgrounds, and such metalinguistic knowledge is not part of PISAS description of reading proficiency. 
On the other hand, the ability to construe figurative language is considered an important constituent 
of interpreting texts, and particularly literary texts. It is recognised that a particular challenge for an 
international assessment of reading is to reflect this ability across languages and cultures. In this item, the 
figurative language in question uses terms ("sea" and "night") that can be regarded as universally familiar, 
and that have a similar connotation across cultures in the context provided by the narrative passage. 

The field trial results indicate that the item had robust psychometric qualities and performed similarly 
across countries and languages. This item demonstrates, then, that it is sometimes possible to successfully 
construct an item that focuses on a text's literary qualities, such as figurative language, for an international 
assessment. This question also demonstrates that while it is most common for multiple-choice items in 
PISA to have four possible response options, sometimes more than four options are given. The item was 
moderately difficult, with less than two-thirds of students answering it correctly. 



PISA 2009 ASSESSMENT FRAMEWORK 



KEY COMPETENCIES IN READING, MATHEMATICS AND SCIENCE 



OECD 2009 



PRINT READING UNIT 9 




African 1>elf 



The Northern Drakensberg Trek: South Africa / 
Lesotho 

Fact file 

OVERVIEW 




o The Northern Drakensberg Trek involves crossing the northern Drakensberg escarpment at high 
altitudes. The route, which is approximately 40 miles (65km) long, straddles the border between 
South Africa and Lesotho, taking 5 strenuous days to compiete. The Trek is fiiled with highlights, 
including breathtaking views over the Amphitheatre to the Devil's Tooth as you make your way to 
the Chain Ladder, and sunrise seen from Mponjwane, which is weli worth setting your alarm for. 

^ Start; The Sentinel car park. Royal Natal National Park. 

^ Finish: The Cathedral Peak Hotel. 



Difficulty and Altitude: This is a high-mountain walk in one of the more remote areas of the 
Drakensberg Range. The going can be quite tough and the days long. Good navigation skills are 
essential for safe crossing. 



TIMING AND SEASONALITY 



^ Best Months to Visit: Aprii, May, June or September, October, November. 

Climate: Summers in the Drakensberg can be very hot and very wet. The winters are much drier, 
but there is always a chance of precipitation, which will probably take the form of snow on the 
high ground. In spring and autumn the daytime temperatures are ideal (between 60°F/1 5°C and 
70°F/20°C), but at night will frequently drop below freezing point. 



Temperature and Precipitation 


Average daiiy maximum temperature 


(°F) 


72 


70 


70 


66 


63 


60 


60 


63 


66 


68 


70 


70 


(°C) 


22 


21 


21 


19 


17 


15 


15 


17 


19 


20 


21 


21 


Average daily minimum temperature 


(°F) 


55 


55 


54 


48 


46 


41 


41 


43 


46 


48 


52 


54 


(°C) 


13 


13 


12 


9 


8 


5 


5 


6 


8 


9 


1 1 


12 


Average monthly precipitation 


(Inches) 


9.3 


8.5 


7.7 


3.1 


1.1 


0.6 


0.5 


1.3 


2.4 


4.0 


6.5 


7.9 


(Millimetres) 


237 


216 


196 


78 


29 


14 


12 


33 


62 


101 


165 


201 




JAN 


FEB 


MAR 


APR 


MAY 


JUN 


JUL 


AUG 


SEPT 


OCT 


NOV 


DEC 
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WALKPROFILE 



LU 

Q 

D 



< 



I 2 000 
(3 660) 



lO 000 
(3 050) 



8 000 
(2 440) 



6 000 
(I 830) 



4000 
(I 220) 







0 

1 




African Trek is classified as mixed in terms of text format, because it contains both continuous elements 
(the prose dot points in the overview) and non-continuous elements (the temperature and precipitation 
table and the walk profile). Since the aim of both the continuous and non-continuous elements of the 
text is to describe the Northern Drakensberg Trek, the text type is description. In terms of situation, the 
text is classified as personal, since it is intended to be read by individuals for their own interest and 
information. 



Question 2: AFRICAN TREK 

At whgt gltitu4e is the Sentinel Qr Pgrki’ Give your answer in feet gn4 metres. 

feet 

metres 

Framework Characteristics 



Situation 


Personal 


Medium 


Print 


Text format 


Non-continuous 


Text type 


Description 


Aspect 


Access and retrieve: Retrieve information 


Question intent 


Locate information that is explicitly stated in a graphic 


Item format 


Closed constructed response 
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Full Credit 

Code 1 : Answers 8000 (feet) AND 2440 (metres). 

No Credit 

Code 0: Other responses. 

■ 8000 metres, 2440 feet. 

■ 6000 metres, 1 830 feet. 

Code 9: Missing. 



This item was easy. It was answered correctly by nearly three-quarters of students. Although the item is 
easy, multiple steps are involved in answering correctly. Firstly, the student must locate the part of the text 
in which the correct answer is found, since the question stem does not direct the student to this part of the 
text. Once the correct part of the text (the Walk Profile) is located, the student must correctly interpret the 
key to the Walk Profile to recognise the units for each number, and finally, retrieve the specified information. 



Question 7: AFRICAN TREK 

Accor4irig to the informgtion provicjed, where wouM you stgy gfter the second d^y of wglkingi’ 

A. At the Sentinel Qr Pgrk. 

B. At I fid i Cgve. 

C. At Rwgnqg Qve. 

D. At Mponjwgne Qve. 

E. At Twins Qve. 

F. At the CgthedrgI Pegk Hotel. 

Framework Characteristics 



Situation 


Personal 


Medium 


Print 


Text format 


Non-continuous 


Text type 


Description 


Aspect 


Access and retrieve: Retrieve information 


Question intent 


Locate information that is explicitly displayed in a mixed format descriptive text 


Item format 


Multiple choice 
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CODING GUIDE 

Full Credit 

Code I : C. At Rwanqa Cave. 

No Credit 

Code 0: Other responses. 

Code 9: Missing. 



This item was difficuit. Just over one-third of students answered correctly. The source of difficulty of this 
item is clearly shown by the pattern of response options chosen: more students chose option B than 
the correct response, option C. To respond correctly, students must understand that the numbers on the 
X-axis of the profile that show each day represent the beginning of that day. Following this logic, the 
place names that appear immediately following each number do not represent the place that walkers 
stay on that day, but rather where they stay on the night before the day represented by each of the 
numbers I to 5. The many students who incorrectly chose option B located the number 2 in the walk 
profile, and assumed that the place name appearing closest to this number would represent the place 
where you would stay on the night of day 2. These students did not fully comprehend the structure of 
this non-continuous text. This kind of careful reading is challenging, but it is an important reading skill. 
This multiple-choice item has six response options, in this case because the six options were naturally 
suggested by the structure of the walk profile, which includes six place names. 



Question 8: AFRICAN TREK 

which 43y ofthe trek 4o you think woul4 be the most 4ibficulti’ Use the information provi4e4 to 
give 9 reason to support your answer. 



Framework Characteristics 



Situation 


Personal 


Medium 


Print 


Text format 


Non-continuous 


Text type 


Description 


Aspect 


Reflect and evaluate; Reflect on and evaluate the content of a text 


Question intent 


Support an opinion using information from a mixed format descriptive text 


Item format 


Open constructed response 



CODING GUIDE 

Full Credit 

Code I : Refers to Day I, Day 2, Day 3, Day 4 OR Day 5 AND gives a reason that is plausible, and 

consistent with the text. Must refer (explicitly or implicitly) to the text. 
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■ Day I . It's uphill all the way. 

■ Day 2. For me, to go up, then down, then up, then down is worse than consistently going up or down. 

■ Day 3. ThereS a horrible peak. 

■ Day 4. That peak looks pretty harsh. 

■ Day 5. It's downhill all the way. 

No Credit 

Code 0: Gives a response that does not refer to the text. 

■ Day I . You won't be used to it yet. [no reference to the text] 

■ Day 5. You'd be absolutely exhausted by theni [no reference to the text] 

Gives a response with no expianation. 

■ Day 2. [no expianation] 

Gives an insufficient or vague response. 

■ Day I . because it would be difficult, [vague] 

■ Day 3. that's the middle, so it would be the most difficult, [insufficient] 

Shows inaccurate comprehension of the materiai or gives an impiausible or irrelevant response. 
Code 9: Missing. 



Like the first two items in this set, this item focuses on the Walk Profile. It requires students to express 
a personal preference in relation to that information and to support their preference using information 
from the text. The need to draw on outside knowledge in expressing a preference means that this item is 
classified as reflect and evaluate. As in other items that require students to express a preference, so long 
as the students support their preference with relevant information from the text, any position may be 
adopted. The item was moderately difficult, with about two-thirds of students answering correctly. 



Question 9: AFRICAN TREK 

On the morning of which 43y ofthe trek woul4 you see the sunrise mentione4 in the overview] 

A. 1 . 

B. Dgy2. 

C. Dgy 3. 

D. Dgy 4. 

E. Dgy 5. 
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Framework Characteristics 



Situation 


Personal 


Medium 


Print 


Text format 


Mixed 


Text type 


Description 


Aspect 


Access and retrieve: Retrieve information 


Question intent 


Locate information by linking information presented in words with information 
presented in a graphic in a mixed format descriptive text 


Item format 


Multiple choice 



CODING GUIDE 

Full Credit 

Code I : D. Day 4. 

No Credit 

Code 0: Other responses. 

Code 9: Missing. 

in this item, the information required is contained in the first dot point of the overview and in the Waik 
Profile. Since students who answer correctly must use information from both the continuous and non- 
continuous parts of this text, the text format for this item is mixed. This item was moderately difficult. It 
was answered correctly by less than two-thirds of students. The difficulty in this item comes from the fact 
that while the end point of the question is to retrieve information that is explicitly stated, students must 
link information from across the text to successfully retrieve the information required. 
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PRINT READING UNIT 1 0 



Metrotransit 
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The Metrotransit unit from the PISA 2009 field trial presents a graphic from a pubiic document, a map of an 
urban transport network in the form of a diagram. It uses fictional place names that participating countries 
were invited to adapt for their national versions. The text is non-continuous; it could be presented as a 
combined list of stations categorised according to line, features and grid locations. Though relatively simple, 
it inciudes a complicating eiement - a key of symbols - the application of which is reguired to gain full credit 
for the question reproduced below. All items related to this unit functioned weil in the field trial. The unit 
was not included in the main survey selection because of concern that students living in rurai areas, who 
would be unfamiliar with urban transport networks, might be disadvantaged. 



Question 1 : METROTRANSIT 

From which Metrotransit station is it possible to tgke both intercity buses gn4 intercity trginsf’ 



Framework Characteristics 



Situation 


Public 


Medium 


Print 


Text format 


Non-continuous 


Text type 


Description 


Aspect 


Access and retrieve: Retrieve information 


Ouestion intent 


Locate information by making links across a map 


Item format 


Closed constructed response 



CODING GUIDE 

Full Credit 

Code 1 : Refers explicitly to Central Station. May mention the line. 

■ Central Station. 

■ Central. 

■ Central (Eastgate). 

■ Central (Line 1 ). 

No Credit 

Code 0: Other responses. 

■ At the station. 

■ Tower. 

■ Line I . 

Code 9: Missing. 



This kind of authentic task exemplifies PISAS emphasis on using reading for practical purposes in everyday 
iife. The item requires students to interpret the map key and apply it to the map to determine which station 
meets the criteria specified in the question. Only one station within the map meets the criteria. The coding 
guide shows that so long as the correct station is unambiguously identified, responses may be expressed in 
a variety of ways. This item was easy. It was answered correctly by over three-quarters of students. 
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Question 2: METROTRANSIT 

if you 3 t Zoo Station gnd you wgnt to go to Stone Bfidge Station, gt which station wouM you 
change linesi’ 

A. Town Hgll. 

B. Rivefside. 

C. B^yside. 

D. 0 I 4 Qugy. 

Framework Characteristics 



Situation 


Public 


Medium 


Print 


Text format 


Non-continuous 


Text type 


Description 


Aspect 


Access and retrieve: Retrieve information 


Question intent 


Identify a junction on a map where there is some competing information 


Item format 


Multiple choice 



CODING GUIDE 

Full Credit 

Code 1 : D. Old Quay. 

No Credit 

Code 0: Other responses. 

Code 9: Missing. 



This question is another example of an authentic task. The item requires students to iocate a specified 
route on the map and in doing so, to determine the point at which it would be necessary to change 
lines. Once the two specified stations have been located, determining the junction in the short route, 
which involves only two of the lines, is a simple task. Over three-quarters of students gained credit 
for this item. 



Question 3: METROTRANSIT 

Some stations like Westend, Zoo gn4 In4epen4ence hgve grey shg4irig groun4 them. Wh^t 4oes the 
shg4ing show gbout these stgtionsi’ 
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Framework Characteristics 



Situation 


Public 


Medium 


Print 


Text format 


Non-continuous 


Text type 


Description 


Aspect 


Reflect and evaluate; Reflect on and evaluate the form of a text 


Question intent 


Identify the purpose of a graphical feature in a map 


Item format 


Short response 



CODING GUIDE 

Question intent 
Full Credit 

Code I : Refers to the fact that these stations are at the end or the start of a line. 

■ End of the line. 

■ Terminus. 



No Credit 

Code 0: Gives an insufficient or vague response. 

Shows inaccurate comprehension of the material or gives an implausible or irrelevant response. 

■ You can get buses there, [possibly, but not what the shading shows] 

■ They are the busiest stations. 

■ They are the biggest stations. 

Code 9: Missing. 



This item was moderately difficult. It was answered correctly by less than two-thirds of students. The item 
requires students to examine a graphical feature (shading) that is common to several stations on the map 
and in doing so, to determine what the shading is intended to represent in the map. The difficulty of this 
question lies in the requirement to generalise about the purpose of a feature. Since this question requires 
students to stand outside the text and consider the way in which it was constructed, the question is 
classified as reflect on and evaluate the form of a text. 



Question 4: METROTRANSIT 

You need to find the shortest route by underground tTil from Sgto Station to Forest Station. 
Prgw on the m^p the route thgt you would tgke. 
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Framework Characteristics 



Situation 


Public 


Medium 


Print 


Text format 


Non-continuous 


Text type 


Description 


Aspect 


Access and retrieve: Retrieve information 


Question intent 


Combine several pieces of information on a map to determine the shortest 
route between two given points 


Item format 


Short response 



Coding guide 

Full Credit 

Code 2: Draws the route shown in the reproduction below; 
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Partial Credit 



Code I : Draws the route shown in the reproduction below: 







No Credit 

Code 0: Other responses. 

Code 9: Missing. 



This access and retrieve task requires students to locate and sequence multiple pieces of information - the 
names of stations - to plan a route. The task simulates a real-life experience that demands careful reading 
of a non-continuous text. The item was constructed to ensure that use of conditional Information - that 
is, information external to the main part of a text - must be processed in order to compiete the task 
successfully. For full credit, students therefore need to use the information provided in the key; that part 
of the system is under construction. Students who take account of this conditional information realise 
that what is apparently the shortest route is currently unusable. The requirement to use conditional 
information increases the difficulty of items significantly. Only about one-third of students in the field trial 
gained full credit for this item. However more than half of the other students, who plotted a route using 
stations on Line 5, were given partial credit, as they demonstrated that in all but one respect (albeit a 
crucial one in practical terms) they had understood and could use information that they had retrieved. 

Most of the items in PISA print reading require a multiple-choice or text response. The short response 
format of this question, involving the drawing of a route on a map, illustrates that the range of response 
types can nevertheless vary to some extent, according to the format that seems most appropriate for the 
particular task. This is bounded, of course, by the practical limitations of a large-scale paper and pen test. 
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ANNEX A2 

Electronic reading sample tasks 



The electronic reading sample tasks are examples of questions 
students answered in the PISA 2009 survey to assess their 
competencies in reading electronic texts. This part of the survey 
was optional. 

All electronic sample tasks may be viewed on the website for 
publicly released items of the PISA electronic reading assessment: 
http://erasq.acer.edu.au. 

username: public 
password: access 



Note that the images of the electronic reading assessment stimuli 
are multicoloured. The true colours as presented on the screen are 
therefore not reflected in this publication. 
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ELECTRONIC READING UNIT 1 



Philosophers' Cafe 




Menu 

Home 

Bentham 



Confucius 



Zeno 



Activities 



Ptilosophers’ Cafe - Hooe - E007P01 - Internet Browser 



http : //WWW. phik>sophers*ca fe . corrVinbex . htrri 



□ □ 



Philosophers' Cafe > Home 



Where did our ideas of how to 
behave come from? 



This unit is based on the idea of a Web Quest, a type of website that typically involves teaching and 
learning activities, including tasks and resources. It therefore fits into the educational context. The 
stimulus for Philosophers' Cafe includes encyclopaedia-style entries on philosophy loosely related to the 
notion of the good life, in a way intended to exploit the visual and interactive possibilities of the ERA 
environment. The screen shot above shows the Home page for the Philosophers' Cafe. On the right is 
a menu with links to information about three philosophers, Zeno, Confucius and Bentham, as well as 
to a series of activities. The central part of the screen shows a cafe scene populated by young people. It 
includes thought bubbles with philosophical questions that visitors to the cafe (or students of philosophy) 
might be considering. It also includes images of the three philosophers, Zeno, Confucius and Bentham. 
Each of the images of philosophers has a hyperlink to the same page of information as the matching 
menu item to the right. 

The screen shot below shows the page that can be accessed by clicking on the menu item Confucius, 
or on the image of Confucius in the top right of the cafe scene. There are similar pages for Zeno and 
Bentham. Each text presents some historical context for the philosopher, and one of his central ideas. It 
is expected that some students would be familiar with the names of one or more of the philosophers, 
but that they would not necessarily know much about his life or beliefs. The context is thus likely to be 
unfamiliar, while the subject matter is considered to be complex, since it deals with highly abstract ideas 
developed in the past, sometimes a very distant past. 
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Task 1: PHILOSOPHERS' CAFE 



Phiosophers' Cafe - Confuaus - E007P05 - Internet Browser 
Q Back I I I Address 



http://www. pMosophers-cafe.corryconfucMS. htrrf 



Philosophers' Caf6 > Confucius 

"Do not do to other people what you do not want them to do to you." 



Confucius was a Chir>ese philosopher who lived from 551 to 
479 BC. His ^ther died when he was three. ar>d he was 
brought up in poverty by his mother. It was a time of chaos 
artd constant war in China. Confl)Cius saw the suffering of 
people arourxl him arxf wanted to try to put an eixf to it. hie 
wanted peace and prosperity for the people. 

Confucius believed that if society were to improve. p>eople 
needed to char>ge their own behaviour, hie thought that this 
was especially Important for rulers, because their behaviour 
had a stror>g influerx:e on the lives of ordinary people. Our 
fundamental aim should be to live in harmony with other 
people. 

According to Cw>fucius. one of the most important ways we 
can achieve harmony is through what he called "Ren", which 
can be translated as "kirKlr>ess"; that is. consideration of or>e 
human beirtg for another. 



FactI 

I There are approximately 6 
million Confucians in the 
world today. 



i. 

t. 




You gt the Philosophefs' Cgfe Home pgge. Click on the link fot Confucius. Wh^t 4i4 Confucius 
megn by "Ren'C 

A Pegce gn4 pfospefity. 

B Living in chgos gn4 wgn 

C The behgviouf of rulefs. 

P Kin4ness to other- people. 

E Living in hgrmony. 



Framework Characteristics 



Situation 


Educational 


Medium 


Electronic 


Environment 


Authored 


Text format 


Multiple 


Text type 


Exposition / description 


Aspect 


Access and retrieve: Retrieve information 


Question intent 


Locate a definition that is directly stated in a short text containing complex or 
unfamiliar information 


Item format 


Multiple choice 
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CODING GUIDE 

Full Credit 

Code 1 : D. Kindness to other people. 

No Credit 

Code 0: Other responses. 

Code 9: Missing. 



About two-thirds of the students in the field trial gained full credit for this task. Both the navigation and 
text processing demands are low. The task requires students to follow the explicit instruction to click on 
one of the links (either the name "Confucius" or his image in the cafe scene) in order to open the page 
describing Confucius. The information required is explicitly stated in the text. Students can read to the 
end of the text to find the term "Ren" contained in the stem, then make a match between the "kindness' 
in the text and in the key. 



Philosophers* Cafe - Scenario 2 - E007P08 - Internet Browser 

Address http;//www.phiosophers-cafe.corTyactwities-scer«rio2.htrrl 



O Back 



□ □ 



t ^ Philosophers' Cafe > Activities > Scenario 2 




Task 2: PHILOSOPHERS' CAFE 

Go to the Activities pgge. Look gt Scenario 2. Which of Zeno's teachings does this comic illustrgtei' 
A We should consider others in order to live hgppily. 

B We should be concerned gbout our gppegrgnce. 

C We should not let our desires control us. 

P We should not try to change the pgst. 
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Framework Characteristics 



Situation 


Educational 


Medium 


Electronic 


Environment 


Authored 


Text format 


Multiple 


Text type 


Exposition / description 


Aspect 


integrate and interpret: Deveiop an interpretation 


Question intent 


Recognise a simiiarity of theme between a complex abstract text and a cartoon 


Item format 


Multiple choice 



CODING GUIDE 

Full Credit 

Code 1 : C. We should not let our desires control us. 

No Credit 

Code 0: Other responses. 

Code 9: Missing. 



This task requires integration of information from two very different texts. Directed by expiicit navigation 
instructions, students follow two links to locate a page with a comic strip. They have to interpret the 
meaning expressed in the comic strip and reiate it to an idea described on the page for the philosopher, 
Zeno. Navigation is more complex than in task 1 , as they need to visit three pages other than the starting 
page. They will very likely need to go back to the comic strip at least once to check their answer, either 
(most efficiently) by clicking on the "back" button, or by using the menu links on the right of each page. 
The options in the four multiple choice aiternatives present four possibie attitudes, only the final two of 
which inciude ideas attributed to Zeno in the text. The iast of the alternatives, however, does not offer a 
plausibie interpretation of the comic. Students who chose one of the first two options wouid quite likely 
have interpreted them in reiation to the ideas expressed in the comic strip oniy, without reference to 
Zeno. The task thus combines a number of fairly explicit navigation steps with a requirement to interpret 
and integrate information across two texts. This combination of demands heips to explain why this item 
was more difficult than the first task, with about half of the students gaining fuli credit. 
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Task 3; PHILOSOPHERS' CAFE 

PMosopl>efs' Cafe - Tmelrw - E007P03 • Internet Browser 

I Q Back I I I Address http://www.phiosophers-cafe.conVactjvities-tireline.htrrt 






U 




DATE PHILOSOPHER 




Menu 

Home 

Bentham 

Confucius 

Zeno 

Activities 




Go to the Activities pgge. Find the Timeline. I7se the infofmgtion in the website to find when egch of 
the three philosophefs CBenthgm, Confucius gnd Zeno) wgs bofn. Then use the dfop down menus in 
the timeline to show when egch philosopheF- wgs bofn. 

Framework Characteristics 



situation 


Educationai 


Medium 


Electronic 


Environment 


Authored 


Text format 


Multiple 


Text type 


Exposition / description 


Aspect 


Access and retrieve: Retrieve information 


Question intent 


Locate several pieces of information that are explicitly stated across multiple sites 


item format 


Complex multiple choice 



CODING GUIDE 

Full Credit 

Code 1 : Selects correct dates for all three philosophers. 

■ Confucius: 600 - 400 BC 

■ Zeno: 400 - 200 BC 

■ Bentham: 1 600 - 1 800 AD 
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No Credit 

Code 0: Other answers. 

Code 9: Missing. 



This task asks students to place the three philosophers in chronological history, on a timeline that runs 
from 1 000 BC to 2000AD, with the present day shown as "Now". Students respond within the simulated 
website, rather than in the task area at the bottom of the screen. The task requires a minimum of six 
navigation steps, imposing a relatively high navigational demand compared to the difficulty of the 
text processing required. It requires students to follow two explicitly described links in order to find the 
timeline where they respond. They then go to the pages for each of the three philosophers and locate 
the dates when each one lived. These are explicitly and prominently stated in the texts, and cover a 
lengthy time span, from Confucius (551 to 479 BC) and Zeno (333 BC to about 264 BC) to Bentham 
(1 749 to 1 832 AD). There are 1 5 drop-down menus available, each covering a period of 200 years, from 
which students need to select the three that correspond to the times when the philosophers lived. If they 
wish to change their answers, they can reset any individual drop-down menu to be blank. 
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ELECTRONIC READING UNIT 2 



Ice Cream 



Ice Cream - Gtobal Search - E008P02 - Internet Browser 
( I I i ii v,.. !:. I A<Wr ^ http://www.globalsearch.com 



GLOBAL 

SEARCH 



□ □ 

1 1 



\ce cream 



Search 



I O Miracle Ice Cream Diet 

The Hollywood actress says she lost 25 pounds in six weeks ... - by stuffing herself with confectioneries such 
as chocolate. Ice-cream and pudding ... "I ate whichever desserts I liked ... 

WWW diet*Wog.cofrrarcf*v«s-'2007/09’30ii'mifacte_jc«_cr#am_diiei.php 



O The Tribune. Chandigarh. India - Ludhiana Stories 

Dr Grewal talked about different milk prodixits especially those of Indian CMigin like kheer. kulfi (a kind of Ice* 
cream), rabri. milk cake, gajrela and sweets which can be ... 

.V I.V w -lfibijoejnOB-COfTy20(Xl20001220rtdh1-htm 



O Thai Menu 

Sweet and sour fish prepared Thai style with steamed Basmati rice ... soup - Tom yam Koong, Prew wan Kai, 
Basmati Rice. Ice cream. ... Ice Cream with chocolate and nuts ... 

w w w .inOenraiB.iv.'Thairmnu.htrTi 



O Food Standards /\Qencv - Committee members 

j ... served in a number of Government departments ... dietary and lifestyle risk factors ... he led the development _ 

I of manv well known brar>ds of ice cream. ... IZ. 

The stimulus for this unit is a set of internet search results, similar to those obtained using a typical search 
engine. It thus represents the kind of information commonly seen in daily life, whether at school, at 
home or in a work context. The results presented vary substantially in both their sources and the kinds of 
information they include. Although search results are typically generated on the basis of terms selected 
by an individual, the types of information included in the results tend to be very broad. The majority of 
the sites referred to in the results are government, commercial and news websites rather than those with 
a personal focus. To take account of this diversity of content this stimulus is categorised as being in the 
public category. The topic chosen, ice cream, is considered to be very familiar to students. 

The screen shot above shows what readers see when they begin the unit. The full stimulus consists of 
a single page of ten search results, of which only the first four are initially visible. The reader needs to 
scroll down to see the full set. The number of search results included in the stimulus thus entails some 
navigation in order to complete the tasks, and also provides what, from an assessment perspective, can 
be regarded as a large number of multiple-choice distractors. 



Task 1 : ICE CREAM 

This pgge shows segrcb results for ice cregm god similar foods from ground the world. Which segrch 
result is most likely to provide g history of ice cregmi’ Click the button next to the link. 
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Framework Characteristics 



Situation 


Public 


Medium 


Electronic 


Environment 


Authored 


Text format 


Non-continuous 


Text type 


Description 


Aspect 


Integrate and interpret: Form a broad understanding 


Question intent 


Identify relevant search result by recognising the main theme of a short description 


Item format 


Multiple choice 



CODING GUIDE 

Full Credit 

Code 1 : Ice Cream - Online Encyclopaedia Articie about ice Cream. 

No Credit 

Code 0: Other answers. 

Code 9: Missing. 



This question asks students to select which of the series of resuits is most iikely to provide information of a 
specific kind, it asks them to form a broad understanding of the likely content of the links in each search 
result, rejecting those nine results that are irrelevant to their needs, and identifying the only relevant 
result. This is the kind of operation needed every time a person uses an internet search engine and 
examines a page of search results. Since the stimulus is on a single page, the only navigation demand 
is scrolling. The key terms in the questions are "ice cream" (the subject of all the results) and "history". 

The first nine results provide links to information about ice cream recipes or menus (from India, Latvia, 
Ireland and Turkey), a personal blog referring to a Hollywood actress, technical information on ice cream, 
and the page for members of a government agency; all of these can be rejected as unlikely to give a 
history of ice cream. To gain credit, students need to recognise that expressions in the final result, "the 
Chinese invented", "oldest evidence in Europe", "the early 20'^' century" and "One of the oldest types of 
ice cream", collectively refer to the history of ice cream, and suggest that this link would provide more 
information on this topic. 

The navigation demand is low compared to the demand imposed by the need to obtain a general 
understanding of each search result. The item was moderate in difficulty, with between half and two- 
thirds of students gaining full credit. 
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Task 2: ICE CREAM 

This pgge shows search fesults fof ice cfegm grid simiUf foods from ground the wofid. You wgnt to 
know if ice cfegm cgn be pgrt of g heglthy diet. Which segfch result is most likely to give gccufgte 
gdvicei' Click the button next to the link. 

Framework Characteristics 



Situation 


Public 


Medium 


Electronic 


Environment 


Authored 


Text format 


Non-continuous 


Text type 


Description 


Aspect 


Reflect and evaluate; Reflect on and evaluate content of text 


Question intent 


Evaluate search results in terms of relevance and credibility/ trustworthiness of information 


Item format 


Multiple choice 



CODING GUIDE 

Full Credit 

Code 1 : National Food Information Centre: The Food Guidelines-Food Label Connection. 

No Credit 

Code 0: Other answers. 

Code 9: Missing. 



This question represents another very typical task faced by users of the Internet, that is, evaluating the 
trustworthiness of the results for a particular purpose, in this case the quality of advice about whether or 
not ice cream may be part of a healthy diet. 

In order to gain credit for this task, students need to evaluate the various links in terms of both relevance 
and likely authoritativeness. This process should lead them to reject results which provide links to recipes 
and menus, which are not relevant to the focus of the research question referred to in the stem. They 
must also reject results that refer to diet but that lack authoritativeness, such as the blog referring to 
the Hollywood actress "stuffing herself with confectioneries"; and results which have some reference to 
potentially credible sources, but are not likely to be relevant, such as Dr Grewal describing varieties of 
milk products. Possibly the strongest distractor is the result for the Food Standards Agency, which has a 
name that sounds both relevant and credible and includes the words "dietary and lifestyle risk factors". 
However, this search result refers only to committee members of this agency, not to health guidelines. 
Students need to recognise that the fifth result links to a plausibly reliable national organisation, "National 
Food Information Centre", and contains not only relevant information on diet recommendations but 
also the kind of technical language one would expect from a reputable organisation of this kind, such 
as "serving size" and "National Dietary Guidelines suggest you eat a diet providing 30 per cent or less of 
calories (joules)". 

This reflection and evaluation task, which relies heavily on text processing and little on navigation, was 
difficult for students in the field trial, with only about one-third selecting the correct search result. 
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ELECTRONIC READING UNIT 3 



Phishing 



Phishing - Intro - EOlOPOl - Internet Browser 



AfWress http://www.phishingeducatk>n.org 



□ □ 



The Online Phishing Resource Site > Home 






iVt-.iI I-- Pl-iisfiru;'' 

Phishing is a form of identity theft. Its purpose is to trick 
people into handing over their personal information. 
People who collect this information may then use it for 
illegal activities such as stealing money from a bank 
account. A phishing e-mail is an e-mail that imitates a 
well-known company, or claims to offer a genuine 
service, and asks the recipient to provide personal 
information. An example is an e-mail that pretends to 
be from a bank, and asks the recipient for login details 
such as a username and password. A common 
technique is to request urgent action to ensure that an 
account remains valid. 

Phishing Is not always easy to identify, because many 
phishing e-mails look genuine. 




M 

]0 



is Phlshiny a Rea' Problem? 

As access to the Internet and the use of e-mail have 
spread, so has phishing. 

Phishing Facts 

Average loss to each person is a victim of successlul 



The material presented in this unit deals with a common problem that arises from living in a technology- 
rich environment: phishing, the attempt to trick people into providing personal information about 
themselves, so that it may be used for criminal gain. The stimulus, entitled "The Online Phishing Resource 
Site", is based on authentic websites that explain this problem, giving suggestions to readers about 
how to identify phishing and how to avoid becoming a victim of phishing. The home page, an extract 
of which is shown in the screen shot above, explains what phishing is and provides some background 
information about the extent of the problem. It is classified as a public text, in that it relates to activities 
and concerns of the larger society (though it may have a personal application). As a kind of online public 
notice, it assumes anonymous contact with the reader. 



Task 1 : PHISHING 

You gre gt the Home pgge ofthe Online Phishing Resource Site. Accor4ing to the information on this 
pgge, which one of the following is g feature of g phishing e-rngih 

A It gsks for personal information. 

B It contains unwanted advertising. 

C It offers g genuine service. 

P It comes from 9 well-known company. 
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Framework Characteristics 



Situation 


Public 


Medium 


Electronic 


Environment 


Authored 


Text format 


Continuous 


Text type 


Exposition 


Aspect 


Access and retrieve: Retrieve Information 


Question intent 


Locate an important component of an explicitly stated definition 


Item format 


Multiple choice 



CODING GUIDE 



Full Credit 

Code 1 : A. It asks for personal information. 

No Credit 

Code 0: Other responses. 

Code 9: Missing. 



No navigation is required for this task, as the paragraph with ali the target information appears on 
the screen when students begin the unit, so the level of difficulty is driven entirely by text processing. 
To answer this question students need to identify that the wording of the key, "asks for personal 
information", is a paraphrase of the idea stated in the second sentence of the definition on phishing, 
"its purpose is to trick peopie into handing over their personal information". The distractors inciude a 
common misperception ("unwanted advertising" is a common feature of the Internet) as well as ideas 
referred to in the text but which are features of iegitimate e-mails, rather than of attempts at phishing. 
About two-thirds of students gained fuli credit for this task. 



Task 2: PHISHING 

How mgny phishing e-mgils gre sent ground the world in gn gvergge monthi’ 
A 1,200. 

B Over 6 billion. 

C About 25,000. 

P 55,000. 
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Framework Characteristics 



Situation 


Public 


Medium 


Electronic 


Environment 


Authored 


Text format 


Continuous 


Text type 


Exposition 


Aspect 


Access and retrieve: Retrieve information 


Question intent 


Identify the reference of a number in a list 


Item format 


Multiple choice 



CODING GUIDE 



Full Credit 

Code 1 : B. Over 6 billion. 

No Credit 

Code 0: Other responses. 

Code 9: Missing. 



This task uses the same page of stimulus as the first task, but in contrast to that task, this one does make 
a minimal navigation demand, as students have to scroll down a short way in order to reveal the four 
facts about phishing under the heading "Is Phishing a Real Problem", each of which includes one of the 
numbers in the four options for this question. Students are cued to the need to scroll down because the 
bottom of the screen they first see contains only part of a sentence. There is a further cue to the relevance 
of this section of the text in the word "average", which occurs in both the question stem and in the text, 
"Average loss to each person ...". Very little text processing is required to identify the answer, which is 
explicitly stated in the sentence, "Phishing e-mails sent world-wide each month; over 6 billion". 

This question was easier than the previous one, with almost three-quarters of students gaining credit for it. 
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Phishing • Recognising Phishing • E010P02D - Internet Browser 



I O Bact 1 



Address ht^r/Zwww.phishingediJcation.org/recognising-^jhishing.htrTri 



□ □ 



Phishing Is a form of identity theft. Its purpose Is to trick 
people Into handing over their personal information. 
People who collect this Information may then use it for 
illegal activities such as stealing money from a bank 
account. A phishing e-mail Is an e-mail that imitates a 
well-known company, or claims to offer a genuine 
service, and asks the recipient to provide personal 
information. An example is an e-mail that pretends to 
be from a bank, and asks the recipient for login details 
such as a username and password. A common 
technique is to request urgent action to ensure that an 
account remains valid. 

Phishing Is not always easy to identify, because many 
phishing e-mails look genuine. 



#1. Sender’s E-mail Address moft 

#2. E-mail Greeting mmmd moral 

#3. Requests for Personal Information fiu«d morm\ 

#4. Links In an E-mall fmd«t 
Links within an e-mail can be faked. The link might say 
"www.BankNorthside.com" but it could take you elsew^re. ft 
could take you to a website that imitates the site it pretends 
to link to. 



[■ Verify your E-mail with Bank Northsid* 



Hk Bm Vkw Insert 



^ R«ply R«^toAII Fofward 


♦ 




To: PISA Sljdent 

From; bank29@pubicmai.oom 







Dear Bank Norbside user, 



This e-mai was sent by be Bank Nortiside server t> veriV your e-mal 
address. 

You mustcofTpiete bis process by dddng on be ink below and entering 
your Bank Norbside Credit/ Debit Card nurrber and password in be 
spaces provided. 

This process is necessary for your protecfon - because some of our 
merrfoers no longer have access fo beir e-mal addresses and we must 
ver% bem. 

To veriV your e-mai address and access your bank account dck on be 
ink below: 



h1bs://Web.Banl(N0fbside.0Dm.au/sianin/dii/strt>te/e-fTial verft.isp 



11 






E 



E 



This task starts on a different page of the phishing website from the first two tasks. The new page gives 
examples of what different features of phishing e-mail messages may look like. As the screen shot above 
shows, on the left of the page are links that allow readers to read more or to hide additional information 
about typical features of phishing e-mail messages, under numbered headings. This represents a different 
kind of navigation to both scrolling and clicking on links that open new pages, since the additional 
information appears (or can be hidden) on the same page. In this screen shot the user has clicked on "#4. 
Links in an E-mail", thereby displaying the explanatory paragraph below; and an example of this feature is 
highlighted in colour at the bottom of the e-mail on the right, "https://web.BankNorthside.com.au/signin/ 
citifi/scripts/e-maii_verify.jsp". 

To answer this question, students need to click on one or more of these numbered links, until they locate 
a match between the question and the information under heading #4. Both navigation and text process- 
ing demands are relatively low, although not insignificant, and nearly two-thirds of students obtained full 
credit for this task. 

Task 3: PHISHING 

which of the following tricks is explained on the "Recognising Phishing" pggei’ 

A The e-mgil gsks the recipient to 4ongte money to g fgke chgrity. 

B The phishing e-mgil installs spywgre on the user's computer. 

C The guthor ofthe e-mgil inserts g fgke link to g fgke website. 

E The e-mgil pretends the recipient hgs won g prize. 



246 



PISA 2009 ASSESSMENT FRAMEWORK - KEY COMPETENCIES IN READING, MATHEMATICS AND SCIENCE © OECD 2009 




ANNEX A2 • ELECTRONIC READING SAMPLE TASKS 



Framework Characteristics 






Situation 


Public 


Medium 


Electronic 


Environment 


Authored 


Text format 


Continuous 


Text type 


Exposition 


Aspect 


Access and retrieve (retrieve information) 


Question intent 


Locate explicitly stated information 


Item format 


Multiple choice 



CODING GUIDE 

Full Credit 

Code 1 : C. The author of the e-mail inserts a fake link to a fake website. 

No Credit 

Code 0: Other responses. 

Code 9: Missing. 
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ANNEX B 

Background questionnaires 



Annex B provides the background questionnaires used in the 
PISA 2009 survey to obtain information about the participating 
schools, students and parents. There are different questionnaires 
for each group: 

The school questionnaire is administered to principals 
and covers the structure and organisation of the school; 
the student and teacher body; the school's resources; 
the school's instruction, curriculum and assessment; the 
school climate; the school's policies and practices; and the 
characteristics of the principal or designate. 

The questionnaire for students addresses their educational 
career; family context and home resources; individual 
engagement in reading; instructional time, learning and 
assessment; classroom and school climate; access to and 
use of libraries; and students' strategies in reading and 
understanding texts. 

Two optional questionnaires are administered to students: 
an educational career questionnaire covering the students' 
educational histories and career aspirations; and a 
questionnaire about students' access to information and 
communication technology and their use of and attitude 
towards computers. 

A third optional questionnaire for parents covers their 
children's past reading engagement; their own reading 
engagement; home reading resources and support; 
parents' background; their perception of and involvement 
in school; and school choice. 
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ANNEX B * BACKGROUND QUESTIONNAIRES 



SCHOOL OUESTIONNAIRE 

The school questionnaire is administered to the school principal and takes about 30 minutes to complete. 
It covers a variety of school-reiated aspects: 

the structure and organisation of the school 

the student and teacher body 

the school's resources 

the school's instruction, curricuium and assessment 

the school climate 

the school policies and practices 

the characteristics of the principal or designate 

Technical terms are given in <brackets> and are adapted to the nationai context by the national data 
collection centre of the participating country or economy. In this annex, an explanation of the technical 
terms is given below the questionnaire item. 



The structure and organisation of the schooi 

Grade levels 

Q1 Do you have the following grade levels in your school? 

( Please tick one box in each row) 



Yes 


No 


a) 


<Grade 1 > 


a, 


a. 


b) 


<Grade 2> 


n, 


a. 


c) 


<Grade 3> 


a, 


a. 


d) 


<Grade 4> 


a, 


a. 


e) 


<Grade 5> 


a, 


a. 


f) 


<Grade 6> 


a, 


a. 


g) 


<Grade 7> 


D, 




h) 


<Grade 8> 


n, 


a. 


i) 


<Grade 9> 


a, 


a. 


j) 


<Grade 1 0> 


D, 




k) 


<Grade 1 1 > 


a, 


a. 


1) 


<Grade 1 2> 


a, 


a. 


m) 


<Grade 1 3> 


Or 


a. 


n) 


<Ungraded school> 


a, 


a. 



Note. 

<Grade> - refers to the administrative level of the student in the school. <Grade I > is the first year of the 
primary level of education, when systematic studies characteristic of primary education begin, e.g. in reading, 
writing and mathematics. <Grade 2> is the year immediately following <Grade I >, etc. 

<Ungraded school> is a school, or a study programme within a school, where there are no grades. 
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Public or private 



Q2 



Is your school a public or a private school? 

( Please tick only one box) 



ANNEX B » BACKGROUND QUESTIONNAIRES 




A public school 

(This is a school managed directly or indirectly by a public education authority, 
government agency, or governing board appointed by government or elected by 
public franchise.) 



A private school 

(This is a school managed directly or indirectly by a non-government 
organisation; e.g. a church, trade union, business, or other private institution.) 



School funding sources 

Q3 About what percentage of your total funding for a typical school year comes from 
the following sources? 

(Please write a number in each row. Write 0 (zero) if no funding comes from that source) 

O/ 

/o 

a) Government (includes departments, local, regional, state and national) 

b) Student fees or school charges paid by parents 

c) Benefactors, donations, bequests, sponsorships, parent fundraising 

d) Other 

Total 100 % 



School location 

Q4 Which of the following definitions best describes the community in which your 
school is located? 

( Please tick only one box) 

A village, hamlet or rural area (fewer than 3 000 people) 

A small town (3 000 to about 1 5 000 people) 

A town (1 5 000 to about 1 00 000 people) Uj 

A city (1 00 000 to about 1 000 000 people) 

A large city (with over 1 000 000 people) 

Competition between schools 

Q5 We are interested in the options parents have when choosing a school for their 

children. Which of the following statements best describes the schooling available to 
students in your location? 

( Please tick only one box) 

There are two or more other schools in this area that compete for our students 
There is one other school in this area that competes for our students 
There are no other schools in this area that compete for our students 
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Compositional school characteristics 

Total school enrolment per gender 

Q6 As at <month day, 2009>, what was the total school enrolment (number of students)? 

(Please write a number in each line. Write 0 (zero) if there are none) 

a) Number of boys: 

b) Number of girls: 

Note. 

<month day, 2009> - This is a date about one month before the data coiiection of the Main Survey starts in 
the country or economy. 



% students repeating grade 

Q7 About what percentage of students in your school repeated a grade, at these <ISCED 
levels>, last academic year? 

(Please write a number in each row. Write 0 (zero) if nobody repeated a grade. Tick 
the 'not available' box if the <ISCED level> does not exist in your school) 



0 / 

/o 


<ISCED level> 
not available in 
this school 


a) 


The approximate percentage of students repeating a 








grade at <ISCED 2> in this school last year was: 






b) 


The approximate percentage of students repeating a 








grade at <ISCED 3> in this school last year was: 




^996 



Note. 

<ISCED level> - This term refers to the internationai Standard Ciassification of Education (iSCED-97) which was 
adopted by the UNESCO Generai Conference in 1 997 (OECD, 1 999). The structure of educationai systems 
varies wideiy between countries, and iSCED-97 provides a muiti-dimensionai framework to coiiect and report 
internationaiiy comparabie data on educationai programmes with a simiiar ievei of educationai content. Six 
iSCED ieveis are defined: 

■ iSCED ievei 0: Pre-primary ievei of education. Should be centre or school-based, be designed to meet 
the educational and development needs of children at least 3 years of age and have staff that are 
adequately trained to provide an educational programme for the children. 

■ ISCED level i : Primary level of education. Beginning of systematic studies characteristic of primary 
education, e.g. reading, writing and mathematics. Entry into the nationally designated primary 
institutions or programmes. 

■ ISCED level 2: Lower secondary level of education. Programmes at the start of level 2 should correspond 
to the point where programmes are beginning to be organised in a more subject-oriented pattern, using 
more specialised teachers conducting classes in their field of specialisation. 

■ ISCED level 3: Upper secondary level of education. The final stage of secondary education in most OECD 
countries. Instruction is often more organised along subject-matter lines than at ISCED level 2 and teachers 
typically need to have a higher level, or more subject-specific, qualification than at ISCED level 2. 

ISCED 3A is designed to provide direct access to ISCED 5A. ISCED 3B is designed to provide direct access 
to ISCED 5B. ISCED 3C is not designed to lead directly to ISCED 5A or 5B. These programmes lead 
directly to labour market, ISCED 4 programmes or other ISCED 3 programmes. 

■ ISCED level 4: Post-secondary non-tertiary education. They are often not significantly more advanced 
than programmes at ISCED level 3 but they serve to broaden the knowledge of participants who have 
already completed a programme at level 3. 
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■ ISCED levels 5/6: Tertiary level of education. 

^ ISCED 5A programmes that are largely theoretically based and are Intended to provide sufficient 
qualifications for gaining entry into advanced research programmes and professions with high skills 
requirements. 

^ ISCED 5B programmes that are generally more practical/technical/occupationally specific than 
ISCED 5A programmes. 

o ISCED level 6 is reserved for tertiary programmes that lead to the award of an advanced research 
qualification. 

REFERENCE 

OECD (1999) Classifying Educational Programmes Manual for ISCED-97 Implementation in OECD Countries. 

Paris: OECD Publications. Available on-line: http://www.oecd.Org/dataoecd/7/2/! 962350.pdf. 

% of immigrant students 

Q8 About how many students in <national modal grade for 1 5-year-olds> in your school 

have a <first language> that is not <the test language>? 

(Please tick only one box.) 



a) 


60% or more 


□ 


b) 


40% or more but less than 60% 


□. 


c) 


20% or more but less than 40% 


□. 


d) 


1 0% or more but less than 20% 


□, 


e) 


More than 0% but less than 1 0% 




f) 


None 





Note. 

<national modal grade for 1 5-year-olds> - This is the actual name of the grade attended by most 1 5-year- 
olds in the participating country or economy. 

<first language> refers to the language first learnt by the student as a child - often referred to as the mother-tongue. 
In administering the questionnaire, <the test language> is replaced with the name of the language of the test. 



Composition and qualification of teaching staff 

Q9 How many of the following are on the staff of your school? 

Include both full-time and part-time teachers. A full-time teacher is employed at least 
90% of the time as a teacher for the full school year. All other teachers should be 
considered part-time. 

(Please write a number in each space provided. Write 0 (zero) if there is none) 

Full-time Part-time 

a) Teachers in TOTAL 

b) Teachers fully certified by <the appropriate authority> 

c) Teachers with an <ISCED5A> qualification 

Note. 

<Appropriate authority> - This is the government agency which is empowered to certify that a person is 
permitted to work as a school teacher. 

<ISCED 5A> typically involves three or four years full time tertiary study (such as a Bachelor’s Degree, for 
example). For a detailed definition of ISCED levels, see the annotation to Q7. 
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The school's resources 

Availability of computers 



Q1 Oa At your school, what is the total number of students in the 
<national modal grade for 15-year-olds>? 

Q1 Ob Approximately, how many computers are available for these 
students for educational purposes? 

Q1 Oc Approximately, how many of these computers are connected to 
the Internet/World Wide Web? 



Number 



Note. 

<national modal grade for 1 5-year-olds> - This is the actual name of the grade attended by most 1 5-year- 
olds In the participating country or economy. 



School resources [e.g. teacher shortage; quality of educational resources) 

Q1 1 Is your school s capacity to provide instruction hindered by any of the following 



issues? 

(Please tick one box in each row) 



Not at 
all 


Very 

little 


To 

some 

extent 


A lot 


a) 


A lack of qualified science teachers 


a, 








b) 


A lack of qualified mathematics teachers 


a, 




°3 




c) 


A lack of qualified <test language> teachers 


a, 




°3 


°4 


d) 


A lack of qualified teachers of other subjects 


a, 




°3 


°4 


e) 


A lack of library staff 


a, 




°3 


°4 


f) 


A lack of other support personnel 


a, 




°3 


°4 


g) 


Shortage or inadequacy of science laboratory 
equipment 




u. 


U 3 


U 4 


h) 


Shortage or inadequacy of instructional 
materials {e.g. textbooks) 


n, 




^3 


□4 


i) 


Shortage or inadequacy of computers for 
instruction 






°3 


^4 


j) 


Lack or inadequacy of Internet connectivity 


u, 


u. 


U 3 


U 4 


k) 


Shortage or inadequacy of computer software 
for instruction 


a, 




°3 


°4 


I) 


Shortage or inadequacy of library materials 


a, 




U 3 


U 4 


m) 


Shortage or inadequacy of audio-visual 
resources 


a, 




°3 


°4 



Note. 

In administering the questionnaire, the term <test language> is adapted to the name of the language used in 
the reading literacy test in the participating country or economy. 
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School curriculum and assessment 

Student grouping by ability 



Q12 Some schools organise instruction differently for students with different abilities. What is 
your school s policy about this for students in <national modal grade for 1 5-year-olds>? 

(Please tick one box in each row.) 







For all 
subjects 


For 

some 

subjects 


Not 
for any 
subject 


a) 


Students are grouped by ability into different classes 


D, 






b) 


Students are grouped by ability within their classes 


D. 


°2 


^3 



Note. 

<national modal grade for 1 5-year-olds> - This is the actual name of the grade attended by most 1 5-year- 
olds in the participating country or economy. 



Extra-curricular school activities 



Q13 


<This academic year>, which of the following activities does your school offer to 
students in the <national modal grade for 1 5-year olds>? 




( Please tick one box in each row) 


Yes 


No 


a) 


Band, orchestra or choir 


a . 


^2 


b) 


School play or school musical 


a , 


^2 


c) 


School yearbook, newspaper or magazine 


a, 


°2 


d) 


Volunteering or service activities, e.g. <national example> 




°2 


e) 


Book club 




°2 


f) 


Debating club or debating activities 




^2 


g) 


School club or school competition for foreign language, 
math or science 


u, 


^2 


h) 


Academic club 


u, 


^2 


i) 


Art club or art activities 




^2 


j) 


Sporting team or sporting activities 


a, 


°2 


k) 


Lectures and/or seminars {e.g. guest speakers such as 
writers or journalists) 


a, 


°2 


I) 


Collaboration with local libraries 


a, 


°2 


m) 


Collaboration with local newspapers 


a, 


°2 


n) 


<Country: specific item> 


a, 


°2 



Note. 

<This academic year> - This term refers to the year of schooling which is not necessarily the calendar year. 
<national modal grade for 1 5-year-olds> - This is the actual name of the grade attended by most 1 5-year- 
olds in the participating country or economy. 
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i 

m 



Curncular options offered to immigrant students 

Q14 Does your school offer any of the following options to students in <national modal 
grade for 1 5-year olds> whose <first language> is not <the test language>? 





(Please tick only one box in each row) 








Yes 


No 


a) 


These students attend regular classes and receive additional periods 
of instruction aimed at developing <test language> skills (e.g. 
reading literacy, grammar, vocabulary, communication) 


u, 




b) 


Before transferring to regular classes, these students attend a 
preparatory programme aimed at developing <test language> skills 
(e.g. reading literacy, grammar, vocabulary, communication) 




u. 


c) 


Before transferring to regular classes, these students receive some 
instruction in school subjects through their <first language> 


u, 




d) 


These students receive significant amounts of instruction in their 
<first language> aimed at developing proficiency in both languages 






e) 


Class size is reduced to cater to the special needs of these students 


a, 


^2 



Note. 

<national modal grade for 1 5-year-olds> - This is the actual name of the grade attended by most 1 5-year- 
olds in the participating country or economy. 

<first language> refers to the language first learnt by the student as a child - often referred to as the mother-tongue. 
In administering the questionnaire, <the test language> is replaced with the name of the language of the test. 



General assessment practices 



Q1 5 Generally, in your school, how often are <national modal grade for 1 5-year-olds> 
students assessed using the following methods? 

(Please tick only one box in each row) 



12 



Never 



times a 
year 



35 times 
a year 



Monthly 



More 
than once 
a month 



a) 

b) 

c) 

d) 



Standardised tests 
Teacher-developed tests 
Teachers' judgmental ratings 
Student <portfolios> 



□ 

U 

U 

u 



^2 

^2 

^2 



^3 

^3 

u. 



u. 






□s 

Us 

Us 

Us 



e) 



Student assignments/ 
projects/homework 



U 



U, 



u. 



u. 



u 



s 



Note. 

<national modal grade for 1 5-year-olds> - This is the actual name of the grade attended by most 1 5-year- 
olds in the participating country or economy. 

<first language> refers to the language first learnt by the student as a child - often referred to as the mother-tongue. 
In administering the questionnaire, <the test language> is replaced with the name of the language of the test. 
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Purposes of assessment 

Q1 6 In your school, are assessments of students in <national modal grade for 1 5-year- 



Note. 





olds> used for any of the following purposes? 

(Please tick only one box in each row) 


Yes 


No 


a) 


To inform parents about their child's progress 


a. 




b) 


To make decisions about students' retention or promotion 


a. 




c) 


To group students for instructional purposes 


a. 




d) 


To compare the school to <district or national> performance 


□, 




e) 


To monitor the school's progress from year to year 






f) 


To make judgements about teachers' effectiveness 






g) 


To identify aspects of instruction or the curriculum that could be improved 






h) 


To compare the school with other schools 







<national modal grade for 1 5-year-olds> - This is the actual name of the grade attended by most 1 5-year- 
olds in the participating country or economy. 



School climate 

Student-related and teacher-related factors affecting school climate 
Q1 7 In your school, to what extent is the learning of students hindered by the following 



phenomenon? 

(Please tick one box in each row.) 







Not at 
all 


Very 

little 


To some 
extent 


A lot 


a) 


Teachers' low expectations of students 










b) 


Student absenteeism 










c) 


Poor student-teacher relations 






°3 


°4 


d) 


Disruption of classes by students 






°3 


°4 


e) 


Teachers not meeting individual students' needs 






°3 


°4 


f) 


Teacher absenteeism 


□ , 




°3 


°4 


g) 


Students skipping classes 


u, 




U 3 


U 4 


h) 


Students lacking respect for teachers 


u, 




U 3 


U 4 


i) 


Staff resisting change 


u, 




U 3 


U 4 


j) 


Student use of alcohol or illegal drugs 


u, 




U 3 


U 4 


k) 


Teachers being too strict with students 


u, 




U 3 


U 4 


I) 


Students intimidating or bullying other students 


□, 




°3 


°4 


m) 


Students not being encouraged to achieve their full 
potential 






°3 


°4 
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Parental achievement pressure 

Q1 8 Which statement below best characterises parental expectations towards your school? 

(Please tick only one box) 

There is constant pressure from many parents, who expect our school to set ^ 

very high academic standards and to have our students achieve them ' 

Pressure on the school to achieve higher academic standards among students ^ 
comes from a minority of parents ^ 

Pressure from parents on the school to achieve higher academic standards y 

among students is largely absent ^ 



School policies and practices 

Student admission policies 



Q19 



a) 

b) 

c) 

d) 

e) 

f) 

g) 



How often are the following factors considered when students are admitted to your 
school? 



(Please tick one box in each row.) 



Never [ Sometimes [ Always 



Residence in a particular area 

Student's record of academic performance 
(including placement tests) 

Recommendation of feeder schools 

Parents' endorsement of the instructional or 
religious philosophy of the school 

Whether the student requires or is interested in a 
special programme 

Preference given to family members of current or 
former students 

Other 



U, 



U, 



u. 



u. 



u. 



u. 



u. 



u.. 



u. 



u. 



u. 



u. 



u. 



u. 



Student transferring policies 

Q20 In your school, how likely is it that a student in <national modal grade for 15-year- 



olds> would be transferred to another school because of the following reasons? 

(Please tick one box in each row) 

If students are never transferred, go to Q2 1 



Not 

likely 


Likely 


Very 

likely 


a) 


Low academic achievement 


U, 


^2 




b) 


High academic achievement 


U, 


U2 


U3 


c) 


Behavioural problems 


u, 


U2 


U3 


d) 


Special learning needs 


u, 


U2 


U3 


e) 


Parents' or guardians' request 


□, 




°3 


f) 


Other 




°2 


°3 



Note. 

<national modal grade for 1 5-year-olds> - This is the actual name of the grade attended by most 1 5-year- 
olds In the participating country or economy. 
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Accountability to parents 

Q21 This set of questions explores aspects of the school s <accountability> to parents. 



(Please tick one box in each row) 



Yes 


No 


a) 


Does your school provide information to parents of students in 
<national modal grade for 1 5-year-olds> on their child's academic 
performance relative to other students in <national modal grade for 
1 5-year-olds> in your school? 




^2 


b) 


Does your school provide information to parents of students in 
<national modal grade for 1 5-year-olds> on their child's academic 
performance relative to national or regional <benchmarks>? 


a . 


^2 


c) 


Does your school provide information to parents on the academic 
performance of students in <national modal grade for 1 5-year-olds> 
as a group relative to students in the same grade in other schools? 




^2 



Note. 

<national modal grade for 1 5-year-olds> - This is the actual name of the grade attended by most 1 5-year- 
olds in the participating country or economy. 

Use of achievement data for accountability 

Q22 In your school, are achievement data used in any of the following <accountahility 
procedures>? 

Achievement data include aggregated school or grade-level test scores or grades, or 





graduation rates. 

(Please tick one box in each row) 


Yes 


No 


a) 


Achievement data are posted publicly (e.g. in the media) 


n . 


°2 


b) 


Achievement data are used in evaluation of the principal's performance 




°2 


c) 


Achievement data are used in evaluation of teachers' performance 


n . 


°2 


d) 


Achievement data are used in decisions about instructional resource 
allocation to the school 


n . 


^2 


e) 


Achievement data are tracked over time by an administrative authority 


a , 


°2 



Teacher evaluation 



Q23 During the last year, have any of the following methods been used to monitor the 





practice of <test language> teachers at your school? 

(Please tick one box in each row.) 


Yes 


No 


a) 


Tests or assessments of student achievement 




^2 


b) 


Teacher peer review (of lesson plans, assessment instruments, lessons) 




^2 


c) 


Principal or senior staff observations of lessons 




^2 


d) 


Observation of classes by inspectors or other persons external to the school 




^2 



Note. 

In administering the questionnaire, <the test language> is replaced with the name of the language of the 
test in the participating country or economy. 



PISA 2009 ASSESSMENT FRAMEWORK - KEY COMPETENCIES IN READING, MATHEMATICS AND SCIENCE C OECD 2009 



ANNEX B » BACKGROUND QUESTIONNAIRES 



School autonomy; responsibility for staffing, budgeting, curriculum and assessment 
Q24 Regarding your school, who has a considerable responsibility for the following tasks? 



(Please tick as many boxes as appropriate in each row) 







Principals 


Teachers 


<School 

governing 

board> 


<Regional 
or local 
education 
authority> 


National 

education 

authority 


a) 


Selecting teachers for hire 


□ , 






n. 


Os 


b) 


Firing teachers 








n. 




c) 


Establishing teachers' starting salaries 






°3 


n. 




d) 


Determining teachers' salaries 
increases 






°3 


n. 




e) 


Formulating the school budget 


U, 


^2 


^3 


u. 




f) 


Deciding on budget allocations 
within the school 


u, 


^2 


U 3 


u. 


Us 


g) 


Establishing student disciplinary 
policies 


u, 


^2 


U 3 


u. 


Us 


h) 


Establishing student assessment 
policies 


□, 


°2 


U 3 


a. 


Us 


i) 


Approving students for admission 
to the school 




°2 


D 3 


a. 


Us 


j) 


Choosing which textbooks are used 




°2 


°3 


a. 


Us 


k) 


Determining course content 




°2 


°3 


n. 


Us 


1) 


Deciding which courses are offered 




°2 


°3 


a. 


Us 



School autonomy; external influences on staffing, budgeting, curriculum and assessment 

Q25 Regarding your school, which of the following bodies exert a direct influence on decision- 
making about staffing, budgeting, instructional content and assessment practices? 

( Please tick as many boxes as apply) 



Area of influence 







Staffing 


Budgeting 


Instructional 

content 


Assessment 

practices 


a) 


Regional or national education 


u, 


u 


u 


u 




authorities (e.g. inspectorates) 


^2 


^3 


4 


b) 


The school's <governing board> 


u, 


U2 


U3 


u. 


c) 


Parent groups 


u, 


U2 


U3 


u. 


d) 


Teacher groups 
(e.g. Staff Association, 


□, 










curriculum committees, trade 


I 


2 


3 


4 


e) 


union) 

Student groups 

(e.g. Student Association, youth 


u, 


U2 


Us 


u. 


f) 


organisation) 

External examination boards 


u, 


U2 


U3 


u. 
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Educational leadership 

Q26 Below you can find statements about your management of this school. Please 



indicate the frequency of the following activities and behaviours in your school 
during the last school year. 

(Please tick only one box in each row) 





Never 


Seldom 


Quite 

often 


Very 

often 


1 make sure that the professional development 
activities of teachers are in accordance with the 


u, 








teaching goals of the school 
1 ensure that teachers work according to the 


a. 


□ 


□ 


□ 


school's educational goals 






^4 


1 observe instruction in classrooms 


a, 


°2 


°3 


°4 


1 use student performance results to develop 


a. 


□ 


□ 


□ 


the school's educational goals 




'-'3 


^4 


1 give teachers suggestions as to how they 


a. 


□ 


□ 


□ 


can improve their teaching 


^2 


3 


4 


1 monitor students' work 


a, 


^2 


°3 


°4 


When a teacher has problems in his/her 
classroom, 1 take the initiative to discuss 




^2 


°3 


°4 


matters 

1 inform teachers about possibilities for 




□ 


□ 


□ 


updating their knowledge and skills 


2 


'-'3 


^4 


1 check to see whether classroom activities 
are in keeping with our educational goals 


u, 


^2 


U3 


U4 


1 take exam results into account in decisions 
regarding curriculum development 


u, 




U3 


U4 


1 ensure that there is clarity concerning the 
responsibility for coordinating the curriculum 


u, 


^2 


U3 


U4 


1 pay attention to disruptive behaviour in 


u, 


□ 


u 


□ 


classrooms 


^2 


'-'3 


4 


1 take over lessons from teachers who are 
unexpectedly absent 


a, 


°2 


°3 


°4 



About you 

Q27 Are you female or male? 

Female Male 



PISA 2009 ASSESSMENT FRAMEWORK - KEY COMPETENCIES IN READING, MATHEMATICS AND SCIENCE © OECD 2009 




ANNEX B * BACKGROUND QUESTIONNAIRES 



STUDENT QUESTIONNAIRE 

The student questionnaire is administered after the literacy assessment and takes students about 30 
minutes to complete. The core questions on home background are similar to those used in the previous 
PISA assessments. The questionnaire covers the following aspects: 

■ student characteristics and educational career 

■ family context and home resources 

■ individual engagement in reading 

M instructional time, learning and assessment 

■ classroom and school climate 

■ access to and use of libraries 

■ student's strategies in reading and understanding texts 

The questions on the strategies in reading and understanding texts are not replicated in this annex. A 
sample item for this scaie is given in Chapter I , metacognition. 

Technicai terms are given in <brackets> and are adapted to the national context by the national data 
collection centre of the participating country or economy, in this annex, an expianation of the technicai 
terms is given below the questionnaire item. 



Student basic characteristics 

Grade 

Q1 What <grade> are you in? 



<Grade> - This term refers to the administrative level of the student in the school. In many countries, the 
number of years in schooling is the usual measure of grade. 

Study programme 



<grade> 



Note. 



Q2 Which one of the following <programmes> are you in? 

(Please tick only one box) 



<Programme 1 > 
<Programme 2> 
<Programme 3> 
<Programme 4> 
<Programme 5> 
<Programme 6> 




Date of birth 

Q3 On what date were you born? 



(Please write the day, month and year you were born) 



Day 



Month 



19 

Year 
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Gender 



Q4 Are you female or male? 



Eemale 


Male 




! °2 



Student's educational career 

ISCED 0 attendance 

Q5 Did you attend <ISCED 0>? 

No □, 

Yes, for one year or less Dj 

Yes, for more than one year Dj 

Note. 

<ISCED 0> - This level refers to pre-primary education defined as the initial stage of organised instruction 
designed primarily to introduce very young children to a school-type environment. Instruction is school-based 
or centre-based, typically for children between 3 and 6 years. For a detailed definition of ISCED levels, see the 
annotation to Q7 of the school questionnaire. 

ISCED 1 starting age 

Q6 How old were you when you started <ISCED 1 >? 

Years 

Note. 

<ISCED ] > - This refers to primary education which begins between age 5 and 7 and generally lasts 4 years {e.g. 
Germany) to 6 years (mode of OECD countries). Primary education is designed to provide a sound basic education 
in reading, writing and mathematics along with an elementary understanding of other subjects. For a detailed 
definition of ISCED levels, see the annotation to Q7 of the school questionnaire. 

Grade repeating 

Q7 Have you ever repeated a <grade>? 





( Please <tick> only one box 


in each row.) 

No, never 


Yes, once 


Yes, twice or more 


a) 


At<ISCED 1 > 


°i 


□2 




b) 


At<ISCED 2 > 


'J| 


^2 




c) 


At<ISCED 3 > 


J| 


^2 


U3 



Note. 

<Grade> - This term refers to the administrative level of the student in the school. In many countries, the 
number of years in schooling is the usual measure of grade. 

For a definition of the <ISCED level>, see the annotation to Q7 of the school questionnaire. 
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Family context and home resources 

Family structure 



Q8 Who usually lives at <home> with you? 

(Please <tick> one box in each row.) 



Yes 



a) 


Mother (including stepmother or foster mother) 


U 


b) 


Eather (including stepfather or foster father) 


U 


c) 


Brother(s) (including stepbrothers) 


u 


d) 


Sister(s) (including stepsisters) 


u 


e) 


Others (e.g. cousin) 


u 



No 

u, 

u, 



Mother's main job 1 

Q9a What is your mother s main job? 

(e.g. school teacher, kitchen-hand, sales manager) 

(If she is not working now, please tell us her last main job) 

Please write in the job title. 

Mother's main job 2 

Q9b What does your mother do in her main job? 

(e.g. teaches high school students, helps the cook prepare meals in a restaurant, 
manages a sales team) 

Please use a sentence to describe the kind of work she does or did in that job. 



Mother's education 

Q1 0 What is the <highest level of schooling> completed by your mother? 

If you are not sure which box to choose, please ask the <test administrator> for help. 
( Please tick only one box) 

<ISCED level 3A> □, 

<ISCED level 3B, 3C> 

<ISCED level 2> ^3 

<ISCED level 1> 

She did not complete <ISCED level 1 > 

Mother's qualifications 

Q1 1 Does your mother have any of the following qualifications? 



If you are not sure how to answer this question, please ask the <test administrator> for help. 
( Please tick one box in each row) 







Yes 


I No 


a) 


<ISCED level 6> 


a, 




b) 


<ISCED level 5A> 


a, 




c) 


<ISCED level 5B> 


n, 




d) I 


<ISCED level 4> 


n, 


I 



Note. 

For a definition of <ISCED level>, see the annotation to Q7 of the school questionnaire. 
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Employment status mother 

Q1 2 What is your mother currently doing? 

(Please <tick> only one box.) 

a) Working full-time <for pay> 

b) Working part-time <for pay> 

c) Not working, but looking for a job 

d) Other (e.g. home duties, retired) 

Father's main job 1 

Q1 3a What is your father s main job? 

(e.g. school teacher, kitchen-hand, sales manager) 

(If he Is not working now, please tell us his last main job) 

Please write in the job title. 

Father's main job 2 

Q1 3b What does your father do in his main job? 

(e.g. teaches high school students, helps the cook prepare meals in a restaurant, 
manages a sales team) 

Please use a sentence to describe the kind of work he does or did in that job. 



Father's education 

Q14 What is the <highest level of schooling> completed by your father? 

If you are not sure how to answer this question, please ask the <test administrator> for help. 
( Please tick only one box) 

<ISCED level 3A> □, 

<ISCED level 3B, 3C> 

<ISCED level 2> 

<ISCED level 1> U 

4 

He did not complete <ISCED level 1 > 

Father's qualifications 

Q1 5 Does your father have any of the following qualifications? 

If you are not sure which box to choose, please ask the <test administrator> for help. 





( Please tick one box in each row) 


Yes 


No 


a) 


<ISCED level 6> 


^1 


^2 


b) 


<ISCED level 5A> 


^1 


^2 


c) 


<ISCED level 5B> 




^2 


d) 


<ISCED level 4> 




^2 



Note. 

For a definition of <ISCED level>, see the annotation to Q7 of the school questionnaire. 
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Employment status father 

Q1 6 What is your father currently doing? 

(Please tick only one box.) 

a) Working full-time <for pay> 

b) Working part-time <for pay> 

c) Not working, but looking for a job 

d) Other (e.g. home duties, retired) 






Country of birth 



Q17 



In what country were you and your parents born? 

(Please tick one answer in each column) 

You 


Mother 


Father 


1 

<Country A> 


Oo, 


Oo, 


°o, 


<Country B> 




^02 


^02 


<Country C> 


^03 


^03 


°03 


<Country D> 




^04 


^04 


<...etc.> 


□ 

<xx> 


□ 

<xx> 


□ 

<xx> 


Other country 


□ 

<xx> 


□ 

<xx> 


□ 

<xx> 



Age at time of arrival 

Q1 8 If you were NOT born in <country of test>, how old were you when you arrived in 
<country of test>? 

If you were less than 12 months old, please write zero (0) 

If you were born in <country of test> please skip this question and gotoQ19 



Years 
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Home language 



Q19 



What language do you speak at home most of the time? 

( Please tick only one box) 



<Language 1 > 
<Language 2> 
<Language 3> 

< ...etc. > 
Other language 



□ 

< 

U 

< 

u 

< 

u 

< 

u 



Home resources 



Q20 Which of the following are in your home? 



( Please tick one box in each row) 







Yes 


No 


a) 


A desk to study at 


u. 


Ua 


b) 


A room of your own 


u, 


Ua 


c) 


A quiet place to study 


u, 


Ua 


d) 


A computer you can use for school work 


u, 


^a 


e) 


Educational software 




°a 


f) 


A link to the Internet 


a, 


°a 


g) 


Classic literature (e.g.. < classical author in the country's language >) 


a, 


°a 


h) 


Books of poetry 


a, 


°a 


i) 


Works of art (e.g. paintings) 


n, 


°a 


j) 


Books to help with your school work 


a, 


°a 


k) 


<Technical reference books> 


n, 


°a 


1) 


A dictionary 


a. 


°a 


m) 


A dishwasher 




°a 


n) 


A <DVD> player 




°a 


o) 


<Country-specific wealth item 1> 




°a 


P) 


<Country-specific wealth item 2> 




Ua 


q) 


<Country-specific wealth item 3> 




Ua 



Family wealth 

Q21 How many of these are there at your home? 



None 


One 


Two 


Three 
or more 


a) 


Cellular phones 


n. 


Oa 






b) 


Televisions 


n, 


Oa 






c) 


Computers 


n. 


°a 


°3 


^4 


d) 


Cars 


n, 


□a 


°3 


^4 


e) 


Rooms with a bath or shower 




Ua 


^3 


U 4 
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Books in the home 
Q22 



How many books are there in your home? 

There are usually about 40 books per metre of shelving. Do not include magazines, 
newspapers, or your schoolbooks. 

( Please tick only one box) 

0-1 0 books □, 



11-25 books 
26-1 00 books 
1 01 -200 books 
201 -500 books 
More than 500 books 



a. 

U3 






Individual engagement in reading 

Reading for enjoyment 

Q23 About how much time do you usually spend reading for enjoyment? 

( Please tick only one box) 

I do not read for enjoyment 

30 minutes or less a day 

More than 30 minutes to less than 60 minutes ^ 

each day ^ 

1 to 2 hours a day 

More than 2 hours a day 

Reading attitude 

Q24 How much do you disagree or agree with these statements about reading? 



(Please tick only one box in each row) 







Strongly 

disagree 


Disagree 


Agree 


Strongly 

agree 


a) 


1 read only if 1 have to 


u. 




U3 


U4 


b) 


Reading is one of my favourite hobbies 


u. 




U3 


U4 


c) 


1 like talking about books with other people 






U3 


U4 


d) 


1 find it hard to finish books 


u. 




U3 


U4 


e) 


1 feel happy if 1 receive a book as a present 






°3 


°4 


f) 


For me, reading is a waste of time 






°3 


°4 


g) 


1 enjoy going to a bookstore or a library 






°3 


^4 


h) 


1 read only to get information that 1 need 






°3 


°4 


i) 


1 cannot sit still and read for more than a 
few minutes 






°3 


°4 


j) 


1 like to express my opinions about books 1 




□ 


□ 


□ 




have read 


^2 


'-'3 


^4 


k) 


1 like to exchange books with my friends 




°2 


°3 


°4 
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Reading diversity 

Q25 How often do you read these materials because you want to? 

(Please tick only one box on each row) 







Never or 


A few 


About 


Several 


Several 






almost 


times a 


once a 


times a 


times a 






never 


year 


month 


month 


week 


a) 


Magazines 


u. 


U 2 




°4 


U.S 


b) 


Comic books 


u, 




Us 


^4 


U.S 


c) 


Fiction (novels, narratives, stories) 






Ua 


^4 


Ua 


d) 


Non-fiction books 






Ua 


^4 


Ua 


e) 


Newspapers 


u, 




Ua 


U 4 


Ua 



Engagement in reading electronic text types 

Q26 How often are you involved in the following reading activities? 

(Please tick only one box on each row. If you don't know what the activity is tick "I 
don 't know what it is ") 



a) 

b) 

c) 

d) 

e) 



g) 



I don t 
know 
what it is 



Reading emails 
<Chaton line> (e.g. <MSN>) 
Reading online news 
Using an online dictionary or 
encyclopaedia (e.g. Wikipedia ) 
Searching online information to 
learn about a particular topic 
Taking part in online group 
discussions or forums 

Searching for practical 
information online (e.g. 
schedules, events, tips, recipes) 



u. 

u, 

a, 




u.. 



u.. 









u. 



u. 









Several 
times a 
week 



U4 

U. 



u. 



u. 









Several 
times a 
day 



^5 

Us 
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Learning by reading strategies 

Learning by reading strategies (eiaboration, memorisation and control) 



Q27 When you are studying, how often do you do the following? 

(Please tick only one box in each row) 







Almost 

never 


Sometimes 


Often 


Almost 

always 


a) 


When 1 study, 1 try to memorize everything 


a, 


n 


n 


n 




that is covered in the text. 




'^3 


^4 


b) 


When 1 study, 1 start by figuring out what 


a, 


n 


n 


n 




exactly 1 need to learn. 


'-'2 


^3 


^4 


c) 


When 1 study, 1 try to memorize as many 




u 


u 


u 




details as possible. 


'-'2 


^3 


^4 


d) 


When 1 study, 1 try to relate new information to 
prior knowledge acquired in other subjects. 




^2 


U3 




e) 


When 1 study, 1 read the text so many times 




□ 


□ 


□ 




that 1 can recite it. 


'-'2 




‘-^4 


f) 


When 1 study, 1 check if 1 understand what 1 


a, 


□ 


□ 


□ 




have read. 


'-'2 


'^3 


4 


g) 


When 1 study, 1 read the text over and over again. 




°2 


°3 




h) 


When 1 study, 1 figure out how the 
information might be useful outside school. 


a, 


°2 


°3 


°4 


i) 


When 1 study, 1 try to figure out which 




1 1 


1 1 


1 1 




concepts 1 still haven't really understood. 


^2 


^3 


^4 


j) 


When 1 study, 1 try to understand the material 
better by relating it to my own experiences. 




^2 


U3 


U4 


k) 


When 1 study, 1 make sure that 1 remember 




□ 


□ 


□ 




the most important points in the text. 


2 


3 


4 


1) 


When 1 study, 1 figure out how the text infor- 
mation fits in with what happens in real life. 


a, 


°2 


^3 


^4 


1) 


When 1 study and 1 don't understand something, 
1 look for additional information to clarify this. 


a, 


°2 


°3 


°4 



Instructional time, learning and assessment 

Length of class period 

Q28 How many minutes, on average, are there in a <class period> for the following 
subjects? 

Number of minutes in a <class period> in <test language> Minutes 

Number of minutes in a <class period> in <mathematics> Minutes 

Number of minutes in a <class period> in <science> Minutes 

Note. 

<class period> - This term refers to the length of time each lesson runs for in a normal school week. 

<test language> - This refers to the language of instruction in which the PISA reading assessment is 
administered. In some countries <test language> may be taught in different school subjects, e.g. English 
language and English literature. If this is the case, <test language> has been adapted accordingly. 
<mathematics> - This consists of all mathematics subjects and courses. 

<science> - This term refers only to the core science subjects of physics, chemistry. Earth science and biology 
either taught in the country's curriculum as separate science subjects, or taught within a single "integrated- 
science" subject. The term does not include related subjects such as engineering, technology, mathematics, 
psychology, economics, nor possible Earth science topics included In geography courses. In many countries 
this term has been adapted to the national context. 
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Number of class periods 

Q29 How many <class periods> per week do you typically have for the following subjects 

Number of class periods per week in <test language> <class periods> 

Number of class periods per week in <mathematics> <class periods> 

Number of class periods per week in <science> <class periods> 




Note. 



See annotations to Q28 of the student questionnaire. 



Total number of class periods 

Q30 In a normal, full week at school, how many <class periods> do you have <in total>? 

Number of ALL <class periods> <class periods> 



Note. 

See annotations to Q28 of the student questionnaire. 



Attendance out-of-school-time lessons 



Q31 


What type of out-of-school-time lessons do you attend currently? 

These are only lessons In subjects that you are also learning at school, that you spend 
learning extra time outside of normal school hours. The lessons may be given at your 
school, at your home or somewhere else. 

(Please tick only one box in each row) 

Yes 1 No 


a) 


<Enrichment lessons> in <test language> 




^2 


b) 


<Enrichment lessons> in <mathematics> 


a, 


^2 


c) 


<Enrichment lessons> in <science> 


a, 




d) 


<Enrichment lessons> in other school subjects 


D, 


°2 


e) 


<Remedial lessons> in <test language> 


n, 




f) 


<Remedial lessons> in <mathematics> 


n, 




g) 


<Remedial lessons> in <science> 


D, 


^2 


h) 


<Remedial lessons> in other subjects 


n, 


D2 


i) 


Eessons to improve your <study skills> 


n, 


°2 



Note. 

<Out-of-school-time lessons> - This term refers to any lessons in the student's school subjects, that he or she 
spends extra time learning outside of normal school hours. The lessons might be held at school, at home, or 
elsewhere. 

<Remedial lessons > - This refers to any lessons in addition to regular lessons designed to help students with 
learning difficulties. 

<Enrichment lessons> - This refers to any lessons in addition to regular lessons designed to extend abilities of 
more able students. 

<Study skjlls> - Study skills are strategies and methods of purposeful learning, usually centred around reading 
and writing, time management, doing homework and/or preparing for tests or exams. 

For the definition of <test language>, <mathematics>, and <science> see the annotations to Q28 in the 
student questionnaire. 
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Time spent on out-of-school-time lessons 

Q32 How many hours do you typically spend per week attending out-of-school-time 
lessons in the following subjects? 

These are only lessons In subjects that you are also learning at school, that you spend 
learning extra time outside of normal school hours. The lessons may be given at your 
school, at your home or somewhere else. 





<test 

language> 


<mathematics> 


<science> 


Other 

subjects 


1 do not attend out-of-school time 
lessons in these subject 


n, 


u, 






Less than 2 hours a week 




u. 


U3 


U3 


2 or more but less than 4 hours a week 




U3 


U3 


U3 


4 or more but less than 6 hours a week 


'J4 


U4 


U4 


U4 


6 or more hours a week 


Us 


Us 


Us 


Us 



Note. 

<Out-of-school-time lessons> are any lessons in the students school subjects, that he or she spends extra 
time learning outside of normal school hours. The lessons might be held at school, at home, or elsewhere. 
For the definition of <test language>, <mathematics>, and <science> see the annotations to Q28 in the 
student questionnaire. 



Classroom and school climate 

Attitude towards school 



Q33 Thinking about what you have learned in school: to what extent do you agree or 
disagree with the following statements? 

(Please tick only one box in each row) 



Strongly 

disagree 



Disagree 



Agree 



Strongly 

agree 



a) 



School has done little to prepare me for 
adult life when I leave school 












□ 



4 



b) School has been a waste of time 

c) School has helped give me confidence to 
make decisions 



























e) 



School has taught me things which could 
be useful in a job 



U. 



U, 



u. 



u. 
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Teacher's interest in student performance 

Q34 How much do you disagree or agree with each of the following statements about 



teachers at your school? 

(Please tick only one box in each row) 



Strongly 

disagree 


Disagree 


Agree 


Strongly 

agree 


a) 


1 get along well with most of my teachers 






°3 


°4 


b) 


Most of my teachers are interested in my 
well-being 




^2 






c) 


Most of my teachers really listen to what 1 
have to say 




^2 




U4 


e) 


If 1 need extra help, 1 will receive it from 
my teachers 




^2 


U3 


U4 


f) 


Most of my teachers treat me fairly 




^2 


U3 


U4 



Class size 

Q35 On average, about how many students attend your <test language> class? 

students 



Note. 

For the definition of <test language> see the annotation to Q28 in the student questionnaire. 

Classroom climate 

Q36 How often do these things happen in your <test language> lessons? 

(Please tick only one box in each row) 







Never or 
hardly 
ever 


In 

some 

lessons 


In most 
lessons 


In all 

lessons 


a) 


Students don't listen to what the teacher 
says 




U2 


U3 


U4 


b) 


There is noise and disorder 


u, 


U2 


U3 


U4 


c) 


The teacher has to wait a long time for 
the students to <quieten down> 




°2 


°3 


°4 


d) 


Students cannot work well 


a, 


°2 


°3 


°4 


e) 


Students don't start working for a long 




n 


n 


n 




time after the lesson begins 


'^2 


'-'3 


^4 



Note. 

For the definition of <test language> see the annotation to Q28 in the student questionnaire. 
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Teacher's reading literacy teaching strategies 
Q37 In your <test language> lessons, how often does the following occur? 







Never or 
hardly 
ever 


In some 
lessons 


In most 
lessons 


In all 

lessons 


a) 


The teacher asks students to explain the 


U, 


U, 


U, 


U, 




meaning of a text 


1 


2 


3 


4 


b) 


The teacher asks questions that challenge 




1 1 


1 1 


1 1 




students to get a better understanding of a text 






^4 


c) 


The teacher gives students enough time to 


a, 


n 


n 


n 




think about their answers 


'-'2 


^3 


^4 


d) 


The teacher recommends a book or author to read 




°2 


°3 




e) 


The teacher encourages students to express 


a, 


□ 


□ 


□ 




their opinion about a text 


'-'2 


^3 


^4 


f) 


The teacher helps students relate the stories 
they read to their lives 


a, 


°2 


°3 




g) 


The teacher shows students how the information 
in texts builds on what they already know 


a, 


°2 


°3 


°4 



Note. 



For the definition of <test language> see the annotation to Q28 in the student questionnaire. 



Teacher's instructional strategies in relation to assignments 
Q38 In your <test language> lessons, how often does the following occur? 



( Please tick only one box in each row) 



Never or 
hardly | In some 
never lessons 



a) 

b) 

c) 

d) 

e) 

f) 

g) 

h) 

i) 



The teacher explains beforehand what is 
expected of the students 

The teacher checks that students are concentrating 
while working on the <reading assignment> 

The teacher discusses students' work, after they 
have finished the <reading assignment> 

The teacher tells students in advance how their 

work is going to be judged 

The teacher asks whether every student has 

understood how to complete the <reading 

assignment> 

The teacher marks students' work 

The teacher gives students the chance to ask 

questions about the <reading assignment> 

The teacher poses questions that motivate 
students to participate actively 
The teacher tells students how well they did on 
the <reading assignment> immediately after 



a, 

a, 

a, 

a, 

n, 

n, 

n, 



u.. 



























In most 
lessons 



U, 



























In all 

lessons 



U, 



























Note. 
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For the definition of <test language>, see the annotation to Q28 in the student questionnaire. 
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Libraries 

Access to and use of a library 

Q39 How often do you visit a library for the following activities? 

( Please tick only one box in each row) 







Never 


A few 
times a 
year 


About 
once a 
month 


Several 
times a 
month 


Several 
times a 
week 


a) 


Borrow books to read for pleasure 


a, 










b) 


Borrow books for school work 


a, 










c) 


Work on homework, course 
assignments or research papers 


D, 




°3 


D 4 


°5 


d) 


Read magazines or newspapers 


D, 




Ob 


□4 


°5 


e) 


Read books for fun 


D, 




Ob 


□4 


Ob 


f) 


Learn about things that are not course- 
related, such as sports, hobbies, people 




u. 


Ub 


U 4 


Ub 


g) 


or music 
Use the Internet 


a, 




^B 


°4 


Ub 



Availability of a school library 
Q40 Does your school have a <school library>? 

No Yes 
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INTERNATIONAL OPTIONS 

As in previous surveys, additionai questionnaire materials were developed and offered as international 
options to the participating countries and economies. In PISA 2009, these are the Educational Career 
Questionnaire and the ICT familiarity questionnaire for students, as well as the parent questionnaire. 

Educational Career Questionnaire 



The Educational Career Questionnaire consists of six questions on the student's interruptions of schooling 
or change of schools, educational aspirations and grade marks, as well as lessons taken out of school. 

The questions refer to ISCED levels, which are explained in the annotation to Q7 of the school 
questionnaire. 



Interruptions during <ISCED 1> 

Q1 Did you ever miss two or more consecutive months of < ISCED 1 >? 

(Please tick only one box) 

No, never 



Yes, once 




Yes, twice or more 


^3 


Interruptions during <ISCED 2> 


Q2 Did you ever miss two or more consecutive months of <ISCED 2> 

(Please tick only one box) 


No, never 




Yes, once 




Yes, twice or more 





Change schools during <ISCED I> 

Q3 Did you change schools when you were attending <ISCED 1>? 

(Please tick only one box) 

No, I attended all of <ISCED 1 > at the same school 

Yes, I changed schools once 

Yes, I changed schools twice or more 



Change schools during <ISCED 2> 

Q4 Did you change schools when you were attending <ISCED 2>? 

(Please tick only one box) 

No, I attended all of <ISCED 2> at the same school 

Yes, I changed schools once 

Yes, I changed schools twice or more 
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Educational aspirations 

Q5 Which of the following do you expect to complete? 

(Please tick as many as apply) 



a) 


<ISCED level 2> 


□. 


b) 


<ISCED level 3B or C> 




c) 


<ISCED level 3A> 


□. 


d) 


<ISCED level 4> 




e) 


<ISCED level 5B> 




f) 


<ISCED level 5A or 6> 





Out-of-school-time lessons during <ISCED 1 > 

Q6 Have you attended the following out-of-school-time lessons during <ISCED 1 >? 

These are lessons in subjects that you learned at school during <ISCED l>, on which 
you spend extra time learning outside of normal school hours. The lessons might have 
been held at your <ISCED 1> school, at your home or somewhere else. 

(Please tick only one box in each row) 



Yes 


No 


a) 


<Enrichment lessons> in <test language> 




^2 


b) 


<Remedial lessons> in <test language> 




^2 


c) 


<Private tutoring> on a <one-to-one> basis in 


1 1 


1 1 




<test language> and/or other subjects 




^2 



Note. 

For the definition of <out-of-school-time lessons>, <enrichment lessons>, and <remedial lessons> see the 

annotations to question Q3 1 in the student questionnaire. 

For the definition of <test language>, see the annotation to question Q28 in the student questionnaire. 



Mark for test language 

Q7 In your last school report, what was your mark in <test language>? 



Note. 

For the definition of <test language>, see the annotation to question Q28 in the student questionnaire. 
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ICT FAMILIARITY QUESTIONNAIRE 

The Information Communication Technology (ICT) familiarity questionnaire consists of questions regarding 
the availability of ICT, and the student's use of, and attitudes towards, computers. It is administered to 
students after the international student questionnaire and takes about five minutes to complete. 



Availability of ICT 

Availability of ICT at home 



Q1 



Is there any of these devices available for you to use at horned 









Yes, but I 








Yes 


don t use it 


No 


a) 


Desktop computer 


a, 






b) 


Portable laptop or notebook 


a, 


a. 




c) 


Internet connection 






°3 


d) 


<Video games console>, e.g. (Sony PlayStation) 






°3 


e) 


Cell phone 






°3 


f) 


Mp3/Mp4 reader, iPod or similar 






°3 


g) 


Printer 


u, 






h) 


USB (memory) stick 






^3 


Availability of ICT at school 








Q2 


Is there any of these devices available for you to use at schooR 








( Please tick one box in each row) 


Yes 


Yes, but I 
don t use it 


No 


a) 


Desktop computer 


u, 


u. 


U 3 


b) 


Portable laptop or notebook 








c) 


Internet connection 






°3 


d) 


Printer 




a. 


°3 


e) 


USB (memory) stick 




a. 


D 3 



General use of computers 

Filter question 

Q3 Have you ever used a computer? 



Yes I 


No 


a, 





If you answered Yes to the above question, please continue. 
If you answered No, please stop here. 
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Use of ICT at home 

Use of the computer at home 

Q4 How often do you use a computer for following activities at horned 

(Please tick one box in each row) 



Never or 
hardly 
ever 


Once or 
twice a 
month 


Once or 
twice a 
week 


Every day 
or almost 
every day 


a) 


Play one-player games 


a, 








b) 


Play collaborative online games 


a, 








c) 


Doing homework on the computer 




a. 


°3 


°4 


d) 


Use e-mail 




a. 


°3 


^4 


e) 


<Chatonline> (e.g. <MSN®> ) 




a. 


°3 


□4 


f) 


Browse the Internet for fun (such as 




n 




n 




watching videos, e.g. < YouTube™ >) 


'-'i 




^3 


‘-^4 


g) 


Download music, films, games or 


n 


n 


n 


n 




software from the Internet 


'-'i 




'-'3 


^4 


h) 


Publish and maintain a personal 


1 1 


1 1 


1 1 


1 1 




website, weblog or blog 




^2 


^3 


^4 


i) 


Participate in online forums, virtual 












communities or spaces 




^2 


U3 


^4 




(e.g. < Second Life® or MySpace™ >) 











Home use of internet and e-mail for school-related tasks 



Q5 How often do you do the following at hornet 

( Please tick one box in each row) 







Never or 
hardly 
ever 


Once or 
twice a 
month 


Once or 
twice a 
week 


Every day 
or almost 
every day 


■ 1 
a) 


Browse the Internet for schoolwork (e.g. 
preparing an essay or presentation) 


u, 


u. 


U3 


U4 


b) 


Use e-mail for communication with other 
pupils about schoolwork 


u, 


u. 


U3 


U4 


c) 


Use e-mail for communication with 
teachers and submissions of homework or 


U, 




U3 


u. 


d) 


other schoolwork 

Download, upload or browse material 
from your school's website (e.g. time table 


U, 


□2 


U3 


□4 


e) 


or course materials) 

Check the school's website for 
announcements, e.g. absence of teachers 


a, 


°2 


°3 


°4 
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Use of ICT at school 



Use of ICT at school 



Q6 How often do you use a computer for following activities at schooP. 

( Please tick one box in each row) 







Never or 
hardly 
ever 


Once or 
twice a 
month 


Once or 
twice a 
week 


Every day 
or almost 
every day 


a) 


<Chat on line> at school 










b) 


Use e-mail at school 


u, 


u. 




U 4 


c) 


Browse the Internet for schoolwork 
Download, upload or browse material 


u, 




U 3 


U 4 


d) 


from the school's website (e.g. 
<intranet>) 


u, 




U 3 


U 4 


e) 


Post your work on the school's 
website 


u. 




U 3 


U 4 


f) 


Play simulations at school 
Practice and drilling, such as 


a, 


a. 


°3 


°4 


g) 


for foreign language learning or 
mathematics 


a, 


a. 




°4 


h) 


Doing individual homework on a 

school computer 

Use school computers for group 


a. 




°3 


°4 


i) 


work and communication with other 
students 


a, 


a. 


°3 


°4 



Use of computers at school in classroom lessons 



Q7 



a) 

b) 

c) 

d) 



In a typical school week, how much time do you spend using the computer during 



classroom lessons? 

(Please tick one box in each row) 



Computer use in <test language> lessons 
Computer use in <mathematics> lessons 
Computer use in <science> lessons 
Computer use in foreign language lessons 



No 

time 


0 30 
minutes 
a week 


31-60 
minutes 
a week 


More 
than 60 
minutes 
a week 


a, 




°3 


°4 








□ 


1 


2 


3 


4 


n, 




^3 


^4 




U 3 


U 3 


U 4 



Note. 

For the definition of <test language>, <mathematics> and <science> see the annotation to question Q28 in 
the student questionnaire. 
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Q8 



To what extent are you able to do each of these tasks on a computer? 

( Please tick one box in each row) 



I can do 
this very 
well by 
myself 



I can do 
this with 
help from 
someone 



I know 
what this 
means but I 
cannot do it 



I don t 
know 
what It 
means 



a) 

b) 

c) 

d) 

e) 



Edit digital photographs or other 
graphic images 
Create a database (e.g. using 
<Microsoft Access®>) 

Use a spreadsheet to plot a graph 

Create a presentation (e.g. using 
<Microsoft Power Point®>) 

Create a multi-media presentation 
(with sound, pictures, video) 



U, 



U, 



u. 



u. 






u. 



u. 



u. 



u. 






u. 



u. 



u. 



u. 



u. 



u. 



u. 



Use of computers at school outside classroom lessons 

Q9 In a typical school week, how much time do you spend using the computer at school 
outside classroom lessons, e.g. in a <school library> or computer room? 

( Please tick only one box) 

I never use the computer at school outside classroom lessons 

About half an hour a week 

About an hour a week 

About two hours a week 

About three hours a week 

About four or more hours a week 



Attitude toward computers 

Attitude toward computers 



Q1 0 Thinking about your experience with computers: To what extent do you agree with 
the following statements? 

( Please tick one box in each row) 



Strongly 

disagree 


Disagree 


Agree 


Strongly 

agree 


a) 


It is very important to me to work with a 
computer 


a, 




°3 




b) 


1 think playing or working with a computer 
is really fun 


a, 








c) 


1 use a computer because 1 am very 
interested 


a, 




^3 


°4 


d) 


1 lose track of time when 1 am working with 
the computer 


n, 




^3 


°4 
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PARENT QUESTIONNAIRE 

One questionnaire is administered per student. The parent questionnaire takes about 20 minutes to 
complete. The parent questionnaire covers parental reports related to several aspects: 

basic parent characteristics 

child's past reading engagement 

parents' own reading engagement 

home reading resources and support 

parents' background 

parents' perception of and involvement in school 
school choice 



Basic parent characteristics 

Respondent 





Who will complete this questionnaire? 






( Please tick all that apply) 




a) 


Mother or other female guardian 


□. 


b) 


Father or other male guardian 


□, 


c) 


Other 


□. 



(If other, please specify) 



Chiid's early reading engagement 

Childcare before <ISCED 0> 

Q2 Did your child participate in 

(Please tick only one box) 



before <ISCED 1 > 

< child care > before <ISCED 0>? 



Yes 


No 







Note. 

<ISCED 0> - This term refers to programs at the pre-primary, or initial stage of organised instruction. Students 
typically enter ISCED Level 0 programs between age 3 and 5. For an explanation of ISCED levels, see the 
annotation to Q7 of the school questionnaire. 
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Child's reading engagement during <ISCED 1 > 

Home reading literacy support beginning <ISCED 1 > 

Q3 When your child attended the first year of <ISCED 1 >, how often did you or 

someone else in your home undertake the following activities with her or him? 

(Please tick only one box in each row) 







Every day 
or almost 
every day 


Once or 
twice a 
week 


Once or 
twice a 
month 


Never or 
almost 
never 


a) 


Read books 


D, 




n .3 




b) 


Tell stories 


n, 






□4 


c) 


Sing songs 


n, 






□4 


d) 


Play with alphabet toys (for example; 
blocks with letters of the alphabet) 


u, 




^3 


U4 


e) 


Talk about things you had done 


u, 




U3 


U4 


f) 


Talk about what you had read 


u, 




U3 


U4 


g) 


Play word games 


u, 


^2 


^3 


^4 


h) 


Write letters or words 


a, 


^2 




°4 


i) 


Read aloud signs and labels 


a, 


°2 


°3 


°4 



Note. 

<ISCED 1 > - This refers to primary education which begins between age 5 and 7 and generally lasts four years 
{e.g. Germany) to six years (most OECD countries). Primary education is designed to provide a sound basic 
education in reading, writing and mathematics along with an elementary understanding of other subjects. For 
a detailed definition of ISCED levels, see the annotation to Q7 of the school questionnaire. 

language beginning <ISCED 1> 

In what language did most of the activities in Question 3 take place? 

(Please tick only one box) 

<test language> 

Another language 

Note. 

For the definition of <test language> see the annotation to Q28 in the student questionnaire. 

Time spent on reading for enjoyment 

Q5 When you are home, how much time do you spend reading for your own enjoyment 
(e.g. magazines, comics, novels, fiction, non-fiction)? 

(Please tick only one box) 

a) More than 1 0 hours a week 

b) 6-1 0 hours a week 

c) 1 -5 hours a week 

d) Less than one hour a week 



Home 

Q4 
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Parent's attitude to reading 

Q6 How much do you disagree or agree with these statements about reading? 

(Please tick only one box in each row.) 



Strongly 

agree 


Agree 


Disagree 


Strongly 

disagree 


a) 


Reading is one of my favourite hobbies 


a, 








b) 


1 feel happy if 1 receive a book as a present 


a, 


a. 






c) 


For me, reading is a waste of time 


a, 




°3 


°4 


d) 


1 enjoy going to a bookstore or a library 


a, 




°3 


°4 



Home reading resources and support 

Home reading resources 



Q7 


Which of the following are available to your child in your home? 

( Please tick one box in each row) 


Yes 1 


No 


a) 


Email 


a, 




b) 


<Chat on line> / <MSN > 


a, 




c) 


Internet connection 


u, 




d) 


Daily newspaper 






e) 


A subscription to a journal or magazine 








Books of his/her very own (do not count school books) 


a, I 


°3 



Home reading literacy support at age 1 5 



Q8 How often do you or someone else in your home do the following things with your child? 

(Please tick only one box on each row) 







Never or 
almost 
never 


Once or 
twice a 
month 


Once or 
twice a 
week 


Every day 
or almost 
every day 


a) 


Discuss political or social issues 




°3 


°3 


°4 


b) 


Discuss books, films or television 
programmes 




°3 


°3 


^4 


c) 


Discuss how well your child is doing 




□ 


□ 


□ 




at school 


^3 


'-'3 


^4 


d) 


Eat <the main meal> with your child 
around a table 


n. 


^3 


°3 


^4 


e) 


Spend time just talking to your child 






°3 


□4 


f) 


Go to a bookstore or library with your 


u, 


u 


u 


u 




child 


^2 


^3 


^4 


g) 


Talk with your child about what he/she 
is reading on his/her own 


u, 


U3 




U4 


h) 


Help your child with his/her 


u, 


I I 


I I 


I I 




homework 


^3 


^3 


^4 
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Parent's educational status 

Father's education 



Q9 


Does the child's father have any of the following qualifications? 

(Please tick one box in each row) 


Yes 


No 


a) 


<ISCED level 5A, 6> 






b) 


<ISCED level 5B> 






c) 


<ISCED level 4> 






d) 


<ISCED level 3A> 




u, 


Note. 








For 


a definition of ISCED levels, see the annotation to Q7 of the school questionnaire. 






Mother's education 






Q10 


Does the child's mother have any of the following qualifications? 

(Please tick one box in each row) 


Yes 


No 


a) 


<ISCED level 5A, 6> 




a. 


b) 


<ISCED level 5B> 






c) 


<ISCED level 4> 




a. 


d) 


<ISCED level 3A> 







Note. 

For a definition of ISCED levels, see the annotation to Q7 of the school questionnaire. 

Parents' economical status 

Annual household income 

Q1 1 What is your annual household income? 

Please add together the total income, before tax, from all members of your household. 
Please remember we ask you to answer questions only if you feel comfortable doing 
so, and that all responses are kept strictly confidential. 

(Please tick only one box.) 

Less than < $A > 

< $A > or more but less than < $B > 

< $B > or more but less than < $C > 

< $C > or more but less than < $D > 

< $D > or more but less than < $E > 

< $E > or more 

6 

Note. 

<$> - This symbol denotes the national currency of the participating country or economy. 

<$C> is a suitably rounded value for the national median household income. For all households In the 
country, the median income is the amount for which half of the households have an income above this 
amount, and half, below. 

<$A> is a suitably rounded value equal to about half of the median household income, <$B> is a value at 
about three quarters of the median household Income, <$D> is a value at about five quarters of the median 
household income, and <$E> is a value at about one-and-a-ha)f times the median household Income. 
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As an example, a country where the median household income is $40 500 would have the following brackets: 
Less than $20 000; 

$20 000 or more but less than $30 000; 

$30 000 or more but less than $40 000; 

$40 000 or more but less than $50 000; 

$50 000 or more but less than $60 000; 

$60 000 or more 



Costs of educational services 

Q12 Please answer the following question thinking just of expenses related to <the student 
who brought this questionnaire home>. 

In the last twelve months, about how much would you have paid to educational 
providers for services? 

In determining this, please include any tuition fees you pay to your child's school, any 
other fees paid to individual teachers in the school or to other teachers for any tutoring 
your child receives, as well as any fees for cram school. 

Do not include the costs of goods like sports equipment, school uniforms, computers 
or textbooks if they are not included in a general fee (that is, if you have to buy these 
things separately). 

( Please tick only one box) 



Nothing 


n, 


More than $0 but less than < $W > 


n. 


< $W or more but less than $X> 


Os 


< $X or more but less than $Y> 




< $Y or more but less than $Z > 


Us 


< $Z > or more 


u, 



Note. 

<$> - This symbol denotes the national currency. 

<Sz> is a suitably rounded value representing an amount of money that could be spent on an expensive 
education, including tutoring etc. About 5% of parents should choose this option in a national sample. 

<SW> Is a suitably rounded value for a public education without any extra tutoring. Where public education is 
absolutely free, this would be set at a low nominal value but above zero {e.g. 50 or 1 00 Euro) to take account 
of incidental fees that might be paid. 

After determining <$Z> and <$W>, equal categories are created between these figures to determine values 

for <$X> and <$Y>. 



Number of children in the household 

Q13 How many children are there in your household (including <the student who 



brought this questionnaire home>? 

(Please tick only one box) 



Yes 


No 


On (;'.e. <the student who brought this questionnaire home>) 




u. 


Two 




u. 


Three 




u. 


Four 


a, 




Five 


n, 


n. 


Six or more 


u, 


u. 
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Parents' perception of and involvement in school 

Parents' perception of school quality 



Q14 How much do you agree or disagree with the following statements? 

(Please tick only one box in each row) 







Strongly 

agree 


Agree 


Disagree 


strongly 

disagree 


a) 


Most of my child's school teachers seem 
competent and dedicated 


a, 


°2 


D3 




b) 


Standards of achievement are high in my 
child's school 


n, 


^2 


□3 


°4 


c) 


1 am happy with the content taught and the 


n, 


n 


n 


n 




instructional methods used in my child's school 






'-'4 


d) 


1 am satisfied with the disciplinary 
atmosphere in my child's school 


u, 


^2 


U3 


U4 


e) 


My child's progress is carefully monitored 


u, 


1 1 


1 1 


1 1 




by the school 


^2 


^3 


^4 


f) 


My child's school provides regular and 


u, 


1 [ 


1 1 


1 1 




useful information on my child's progress 


^2 


^3 


^4 


g) 


My child's school does a good job in 


u, 


1 1 


1 1 


1 1 




educating students 


^2 


^3 


^4 



Parents' involvement in their child's school 

Q1 5 The last <academic year>, have you participated in any of the following school- 



related activities? 

(Please tick one box in each row) 



Yes 


No 


a) 


Discuss your child's behaviour or progress with a teacher 
on your own initiative 

Discuss your child's behaviour or progress on the initiative 
of one of your child's teachers 

Volunteer in physical activities, e.g. building maintenance, 
carpentry, gardening or yard work 
Volunteer in extra-curricular activities, e.g. book club, 
school play, sports, field trip 


1 

a, 


^2 

^2 


b) 


D, 


c) 


a, 


^2 


d) 


D, 


^2 


e) 


Volunteer in the school library or media centre 


n, 


^2 


f) 


<Assist a teacher in the school> 


n, 


^2 


g) 


Appear as a guest speaker 


D, 


°2 


h) 


Participate in local school <government>, e.g. parent 


D, 


□ 


counsel or school management committee 


'-'2 



Note. 

<Last academic year> - refers to the previous year of schooling, which is not necessarily the last calendar year. 
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School choice 

Choice between competing schools 
Q16 

We are interested in the options parents had when choosing the school your child is 
currently attending. 

Which of the following statements best describes the schooling available to students 
in your location? 

(Please tick only one box) 

There are two or more other schools in this area that compete with the school 
my child is currently attending 

There is one other school in this area that competes with the school my child is 
currently attending 

There are no other schools in this area that compete with the school my child is 
currently attending 

Reasons for school choice 

Q1 7 How important are each of the following reasons for choosing a school for your 
child? 

(Please tick only one box in each row) 







Not 


Somewhat 




Very 






important 


important 


Important 


important 


a) 


The school is at a short distance to home 




n. 




°4 


b) 


The school has a good reputation 




u. 


U 3 




c) 


The school offers particular courses or 
school subjects 




u. 


^3 


U 4 


d) 


The school adheres to a particular 
<religious philosophy> 


u, 


u. 


^3 


U 4 


e) 


The school has a particular pedagogical- 
didactical approach, e.g. < example> 


u, 




°3 


U 4 


f) 


Other family members attended the school 


a, 


a. 


°3 


°4 


g) 


<Expenses are low> (e.g. tuition, books, 
room and board) 


D, 


a. 


°3 


°4 


h) 


The school has financial aid available, such 
as a school loan, scholarship, or grant 


D, 


a. 


°3 


°4 


i) 


The school has an active and pleasant 
school climate 


a, 


a. 


°3 


°4 


j) 


The academic achievements of students in 
the school are high 


D, 


a. 


^3 


°4 
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ANNEX C 

PISA expert groups 



Annex C lists the experts who were involved in creating and 
implementing the PISA 2009 survey. They specialise in reading, 
mathematics and science, as well as questionnaire development 
and carrying out large-scale education-related international surveys. 
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Reading Expert Group 
Irwin Kirsch - Chair 

Center for Global Assessment 
Educational Testing Service 
Princeton, NJ, United States of America 



Eduardo Vidal- Abarca 

Department of Developmental and Educational Psychology 
Facultad de Psicologia 
Universidad de Valencia 
Valencia, Spain 



Sachiko Adachi 

Faculty of Education, Education and Human Sciences 
Niigata University 
Niigata, Japan 

Charles Alderson 

Department of Linguistics and English Language 
Rowland College 
Lancaster University 
Lancaster, United Kingdom 

John de Jong 

Language Testing Services 
Velp, Netherlands 

John Guthrie 

jean Mullan Professor of Literacy 
Department of Human Development 
University of Maryland 
College Park, MD, United States of America 

Dominique Lafontaine 

Analyse des systemes et des pratiques d’enseignement 
Universite de Liege 
Liege, Belgium 

Minwoo Nam 

Korea Institute of Curriculum & Evaluation 
Seoul, Republic of Korea 

Jean-Francois Rouet 

National Center for Scientific Research 
Laboratoire Langage, Memoire et Developpement Cognitif 
Universite de Poitiers 
Poitiers, France 

Wolfgang Schnotz 

Dept, of General and Educational Psychology 
Faculty of Psychology 
University of Koblenz- Landau 
Landau, Germany 



Mathematics Expert Group 
Jan de Lange - Chair 

Freudenthal Institute 
Utrecht University 
Utrecht, Netherlands 

Werner Blum 

Department of Mathematics 
University of Kassel 
Kassel, Germany 

John Dossey 

Illinois State University 

Eureka, IL, United States of America 

Zbigniew Marciniak 

Institute of Mathematics 
Warsaw University 
Warsaw, Poland 

Mogens Niss 

Professor of Mathematics and Mathematics Education 
Roskilde University 
Roskilde, Denmark 

Yoshinori Shimizu 

Graduate School of Comprehensive Human Sciences 
University ofTsukuba 
Tsukuba, Ibaraki, Japan 

Science Expert Group 

Rodger Bybee - Chair 

Biological Sciences Curriculum Study 
Colorado Springs, CO, United States of America 

Peter Fensham 

School of Mathematics, Science and Technology Education 
Queensland University of Technology 
Annerley, Queensland, Australia 
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Svein Lie 

Department ofTeacher Education and School Development 
(ILS) 

University of Oslo 
Oslo, Norway 

Yasushi Ogura 

Department for International Research and Co-operation 
National Institute for Educational Policy Research of 
Japan 

Tokyo, japan 

Manfred Prenzel 

Leibniz-Institute for Science Education 
University of Kiel 
Kiel, Germany 

Andree Tiberghien 

Universite de Lyon 
Lyon, Erance 

Questionnaire Expert Group 

Jaap Scheerens - Chair 

Eaculty of Behavioural Sciences 

Department of Educational Organisation and Management 
University of Twente 
Enschede, Netherlands 

Pascal Bressoux 

Departement des Sciences de I’Education 
Universite Pierre Mendes France 
Grenoble, France 

Yin Cheong Cheng 

Office ofVice President (Research & Development) 

Hong Kong Institute of Education 
Hong Kong, Hong Kong-China 

David Kaplan 

Department of Educational Psychology 
University ofWisconsin - Madison 
Madison, WI, United States of America 

Eckhard Klieme 

German Institute for International Educational Research (DIPF) 
Department of Educational Quality and Evaluation 
Frankfurt am Main, Germany 



Henry Levin 

Teachers College 

Department of International andTranscultural Studies 

Columbia University 

New York, NY, United States of America 

Pirjo Linnakyla 

Institute for Educational Research 
University of Jyvaskyla 
Jyvaskyla, Finland 

Ludger WoBmann 

IFO Institute for Economic Research 
University of Munich 
Munich, Germany 

Technical Advisory Group 

Keith Rust - Chair 

WESTAT 

Rockville, MD, United States of America 

Ray Adams 

ACER 

Camberwell, Victoria, Australia 

John de Jong 

Language Testing Services 
Velp, Netherlands 
Aletta Grisay 
Consultant 

Saint-Maurice, France 

David Kaplan 

Department of Educational Psychology 
University ofWisconsin - Madison 
Madison, WI, United States of America 

Christian Monseur 

Departement Education 
Universite de Liege 
Liege, Belgium 

Thierry Rocher 

Direction de I’Evaluation, de la Prospective et de la 
Performance 

Ministere de P Education Nationale 
Paris, France 
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Norman Verhelst 

Institute for Educational Measurement 
CITO 



Arnhem, Netherlands 



Kentaro Yamamoto 

Center for Global Assessment 
Educational Testing Service 
Princeton, NJ, United States of America 

Rebecca Zwick 

Gevirtz Graduate School of Education 
University of California at Santa Barbara 
Santa Barbara, CA, United States of America 



Cees Glas 

Institute for Behavioral Research 

Department of Research Methodology, Measurement, 

and Data Analysis 

Eaculty of Behavioural Science 

University ofTwente 

Enschede, Netherlands 

Sophia Rabe-Hesketh 

Graduate School of Education 
University of California, Berkeley 
Berkeley, CA, United States of America 
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